Page History

airflowLocalSettings

airflow_local_settings file as a string (templated). You can bake an airflow_local_settings.py into your image instead. In that case, set this value to null.

See values.yaml

keep

Keep

airflowVersion

Airflow version (Used to make some decisions based on Airflow Version being deployed).

3.1.7

keep

Keep

apiSecretAnnotations

Annotations to add to the Api secret.

{}

Description remains from v2 terms. 

apiSecretKey

The Flask secret key for Airflow Api to encrypt browser session.

~

Description remains from v2 terms. 

apiServer.apiServerConfig

This string (templated) will be mounted into the Airflow API Server as a custom webserver_config.py. You can bake a webserver_config.py in to your image instead or specify a configmap containing the webserver_config.py.

~

Examples:

apiServerConfig: |-
  from airflow import configuration as conf

  # The SQLAlchemy connection string.
  SQLALCHEMY_DATABASE_URI = conf.get('database', 'SQL_ALCHEMY_CONN')

  # Flask-WTF flag for CSRF
  CSRF_ENABLED = True

This is still relevant if the user is using providers-fab and adding more fine-grained authentication integration through fab. That also generalised to BaseAuthManager and to providers that implement 3rd-party libraries and tools. I think the description should be updated.

webserver_config.py migration check mentioned above.

apiServer.apiServerConfigConfigMapName

The configmap name containing the webserver_config.py.

~

Examples:

apiServerConfigConfigMapName: my-api-server-configmap

keep

keep

config

Settings to go into the mounted airflow.cfg

See values.yaml

keep

keep

createUserJob.defaultUser.enabled

Enable default user creation.

True

keep 

defaultAirflowDigest

Default airflow digest to deploy. Overrides tag.

~

defaultAirflowRepository

Default airflow repository. Overrides all the specific images below.

apache/airflow

keep

keep

defaultAirflowTag

Default airflow tag to deploy.

3.1.7

keep

keep

executor

Airflow executor.

CeleryExecutor

keep

fernetKey

The Fernet key used to encrypt passwords (can only be set during install, not upgrade).

~

keep

fernetKeySecretAnnotations

Annotations to add to the Fernet Key secret.

{}

keep

jwtSecret

Secret key used to encode and decode JWTs to authenticate to public and private APIs (can only be set during install, not upgrade).

~

keep

jwtSecretAnnotations

Annotations to add to the JWT secret.

{}

keep

pgbouncer.enabled

Enable PgBouncer.

False

keep

pgbouncer.mountConfigSecret

Whether to mount the config secret files under /etc/pgbouncer/ by default.

True

keep

schedulerName

Specify kube scheduler name for Pods.

~

webserver.webserverConfig

This string (templated) will be mounted into the Airflow webserver as a custom webserver_config.py. You can bake a webserver_config.py in to your image instead or specify a configmap containing the webserver_config.py.

~

Examples:

webserverConfig: |-
  from airflow import configuration as conf

  # The SQLAlchemy connection string.
  SQLALCHEMY_DATABASE_URI = conf.get('database', 'SQL_ALCHEMY_CONN')

  # Flask-WTF flag for CSRF
  CSRF_ENABLED = True

webserver.webserverConfigConfigMapName

The configmap name containing the webserver_config.py.

~

Examples:

webserverConfigConfigMapName: my-webserver-configmap

webserverSecretAnnotations

Annotations to add to the webserver secret.

{}

webserverSecretKey

The Flask secret key for Airflow Webserver to encrypt browser session.

~

workers.schedulerName

Specify kube scheduler name for Airflow Celery workers objects and pods created with pod-template-file.

~

airflowHome

Airflow home directory. Used for mount paths.

/opt/airflow

keep

keep

allowJobLaunching

Whether various Airflow components launch jobs.

False

It can be automated indeed. We can map to both a feature and a deprecation along with it.

allowPodLaunching

Whether various Airflow components launch pods.

True

It can be automated indeed. We can map to both a feature and a deprecation along with it.

apiSecretKeySecretName

The Secret name containing Flask secret_key for the Api.

~

Description remains from v2 terms. 

dags.gitSync.branch

Git branch

v2-2-stable

It makes sense. There are multiple ways in various systems of mounting the dags into the components. Making it bundled can help with easier additions on top of it for mounting operations.

dags.gitSync.containerName

Git sync container name.

git-sync

dags.gitSync.credentialsSecret

Name of a Secret containing the repo GIT_SYNC_USERNAME and GIT_SYNC_PASSWORD.

~

dags.gitSync.depth

Repository depth.

1

dags.gitSync.emptyDirConfig

Configuration for dags empty dir volume.

~

dags.gitSync.enabled

Enable Git sync.

False

dags.gitSync.env

Environment variables for git sync container.

[]

Examples:

env:
- name: GIT_SYNC_TIMEOUT
  value: '60'
env:
- name: GIT_SYNC_USERNAME
  valueFrom:
    secretKeyRef:
      key: username
      name: git-secret

dags.gitSync.envFrom

Extra envFrom ‘items’ that will be added to the definition of Airflow gitSync containers; a string or array are expected (templated).

~

Examples:

envFrom: |-
  - secretRef:
      name: 'proxy-config
envFrom: |-
  - configMapRef:
      name: 'proxy-config

dags.gitSync.extraVolumeMounts

Mount additional volumes into git sync container.

[]

dags.gitSync.knownHosts

When using a ssh private key, the contents of your known_hosts file.

~

Examples:

knownHosts: |-
  <host1>,<ip1> <key1>
  <host2>,<ip2> <key2>
knownHosts: <host1>,<ip1> <key1>

dags.gitSync.maxFailures

The number of consecutive failures allowed before aborting.

0

dags.gitSync.period

Interval between git sync attempts in Go-style duration string. High values are more likely to cause DAGs to become out of sync between different components. Low values cause more traffic to the remote git repository.

5s

dags.gitSync.ref

Git revision branch, tag, or hash.

v2-2-stable

dags.gitSync.repo

Git repository.

https://github.com/apache/airflow.git

dags.gitSync.resources

Resources on workers git-sync sidecar

{}

Examples:

resources:
  limits:
    cpu: 100m
    memory: 128Mi
  requests:
    cpu: 100m
    memory: 128Mi

dags.gitSync.rev

Git revision.

HEAD

dags.gitSync.securityContext

Security context for the gitSync container (deprecated, use securityContexts instead). If not set, the values from securityContext will be used.

{}

Examples:

securityContext:
  runAsGroup: 0
  runAsUser: 50000

dags.gitSync.sshKey

SSH private key

~

dags.gitSync.sshKeySecret

Name of a Secret containing the repo sshKeySecret.

~

dags.gitSync.subPath

Subpath within the repo where dags are located.

tests/dags

dags.gitSync.uid

Git sync container run as user parameter.

65533

dags.gitSync.wait

Interval between git sync attempts in seconds. High values are more likely to cause DAGs to become out of sync between different components. Low values cause more traffic to the remote git repository.

~

dags.mountPath

Where dags volume will be mounted. Works for both persistence and gitSync. If not specified, dags mount path will be set to $AIRFLOW_HOME/dags

~

keep

dags.persistence.accessMode

Access mode of the persistent volume.

ReadWriteOnce

 keep

dags.persistence.annotations

Annotations for the dag PVC

{}

 keep

dags.persistence.enabled

Enable persistent volume for storing dags.

False

 keep

dags.persistence.existingClaim

The name of an existing PVC to use.

~

 keep

dags.persistence.size

Volume size for dags.

1Gi

 keep

dags.persistence.storageClassName

If using a custom StorageClass, pass name here (templated).

~

 keep

dags.persistence.subPath

Subpath within the PVC where dags are located.

~

 keep

elasticsearch.connection

Elasticsearch connection configuration.

{}

Examples:

connection:
  host: '...'
  pass: '...'
  port: '...'
  scheme: https
  user: '...'

elasticsearch.connection.host

Host

""

elasticsearch.connection.pass

Password

""

elasticsearch.connection.port

Port

80

elasticsearch.connection.scheme

Scheme

http

elasticsearch.connection.user

Username

""

elasticsearch.enabled

Enable Elasticsearch task logging.

False

elasticsearch.secretAnnotations

Extra annotations to apply to the elasticsearch secret.

{}

elasticsearch.secretName

A secret containing the connection string.

~

enableBuiltInSecretEnvVars.AIRFLOW_CONN_AIRFLOW_DB

Enable AIRFLOW_CONN_AIRFLOW_DB variable to be read from the Metadata Secret

True

SQLAlchemy connection string from K8s secret. Keeping with a better description and maybe renaming to reflect better. 

enableBuiltInSecretEnvVars.AIRFLOW__API_AUTH__JWT_SECRET

Enable AIRFLOW__API_AUTH__JWT_SECRET variable to be read from the JWT Secret

True

Enable using K8S secret in env var to pass jwt_secret config

  keep

enableBuiltInSecretEnvVars.AIRFLOW__API__SECRET_KEY

Enable AIRFLOW__API__SECRET_KEY variable to be read from the Api Secret Key Secret

True

Similar to above

secret_key

  keep

enableBuiltInSecretEnvVars.AIRFLOW__CELERY__BROKER_URL

Enable AIRFLOW__CELERY__BROKER_URL variable to be read from the Celery Broker URL Secret

True

All these are the same as above, as setting some config that enables to read secret to pass to the env var. Maybe move them to Kustomize? 

enableBuiltInSecretEnvVars.AIRFLOW__CELERY__CELERY_RESULT_BACKEND

Enable AIRFLOW__CELERY__CELERY_RESULT_BACKEND variable to be read from the Celery Result Backend Secret - Airflow 1.10.* variant

True

enableBuiltInSecretEnvVars.AIRFLOW__CELERY__RESULT_BACKEND

Enable AIRFLOW__CELERY__RESULT_BACKEND variable to be read from the Celery Result Backend Secret

True

enableBuiltInSecretEnvVars.AIRFLOW__CORE__FERNET_KEY

Enable AIRFLOW__CORE__FERNET_KEY variable to be read from the Fernet key Secret

True

keep

enableBuiltInSecretEnvVars.AIRFLOW__CORE__SQL_ALCHEMY_CONN

Enable AIRFLOW__CORE__SQL_ALCHEMY_CONN variable to be read from the Metadata Secret

True

enableBuiltInSecretEnvVars.AIRFLOW__DATABASE__SQL_ALCHEMY_CONN

Enable AIRFLOW__DATABASE__SQL_ALCHEMY_CONN variable to be read from the Metadata Secret

True

keep

enableBuiltInSecretEnvVars.AIRFLOW__ELASTICSEARCH__ELASTICSEARCH_HOST

Enable AIRFLOW__ELASTICSEARCH__ELASTICSEARCH_HOST variable to be read from the Elasticsearch Host Secret - Airflow <1.10.4 variant

True

enableBuiltInSecretEnvVars.AIRFLOW__ELASTICSEARCH__HOST

Enable AIRFLOW__ELASTICSEARCH__HOST variable to be read from the Elasticsearch Host Secret

True

enableBuiltInSecretEnvVars.AIRFLOW__OPENSEARCH__HOST

Enable AIRFLOW__OPENSEARCH__HOST variable to be read from the OpenSearch Host Secret

True

enableBuiltInSecretEnvVars.AIRFLOW__WEBSERVER__SECRET_KEY

Enable AIRFLOW__WEBSERVER__SECRET_KEY variable to be read from the Webserver Secret Key Secret

True

env

Environment variables for all Airflow containers.

[]

Examples:

env:
- name: MYENVVAR
  value: something_fun

keep

extraEnv

Extra env ‘items’ that will be added to the definition of Airflow containers; a string is expected (templated).

~

Examples:

extraEnv: |-
  - name: AIRFLOW__CORE__LOAD_EXAMPLES
     value: True

keep

extraEnvFrom

Extra envFrom ‘items’ that will be added to the definition of Airflow containers; a string is expected (templated).

~

Examples:

extraEnvFrom: |-
  - secretRef:
      name: '{{ .Release.Name }}-airflow-connections'
extraEnvFrom: |-
  - configMapRef:
      name: '{{ .Release.Name }}-airflow-variables'

keep

fernetKeySecretName

The Fernet key secret name.

~

gid

Group of airflow user.

0

jwtSecretName

The JWT secret name.

~

logs.emptyDirConfig

Configuration for logs empty dir volume.

~

 keep

logs.persistence.annotations

Annotations to add to logs PVC

{}

 keep

logs.persistence.enabled

Enable persistent volume for storing logs.

False

 keep

logs.persistence.existingClaim

The name of an existing PVC to use.

~

 keep

logs.persistence.size

Volume size for logs.

100Gi

 keep

logs.persistence.storageClassName

If using a custom StorageClass, pass name here (templated).

~

 keep

logs.persistence.subPath

The subpath of the existing PVC to use.

~

 keep

multiNamespaceMode

Whether Airflow can launch workers and/or pods in multiple namespaces. If true, it creates ClusterRole/ClusterRolebinding (with access to entire cluster)

False

opensearch.connection

OpenSearch connection configuration.

{}

Examples:

connection:
  host: '...'
  pass: '...'
  port: '...'
  scheme: https
  user: '...'

opensearch.connection.host

Host

""

opensearch.connection.pass

Password

""

opensearch.connection.port

Port

80

opensearch.connection.scheme

Scheme

http

opensearch.connection.user

Username

""

opensearch.enabled

Enable OpenSearch task logging.

False

opensearch.secretName

A secret containing the connection string.

~

podTemplate

The content of pod_template_file.yaml used for KubernetesExecutor workers (templated). The default (see files/pod-template-file.kubernetes-helm-yaml) already takes into account normal workers configuration parameters (e.g. workers.resources), so you normally won’t need to override this directly.

~

Examples:

podTemplate: |-
  apiVersion: v1
  kind: Pod
  metadata:
    name: placeholder-name
    labels:
      tier: airflow
      component: worker
      release: {{ .Release.Name }}
  spec:
    priorityClassName: high-priority
    containers:
      - name: base
      ...

secret

Secrets for all Airflow containers.

[]

Examples:

secret:
- envName: SecretEnvVar
  secretKey: somekey
  secretName: somesecret

keep

uid

User of airflow user.

50000

volumeMounts

VolumeMounts for all Airflow containers.

[]

keep

volumes

Volumes for all Airflow containers.

[]

keep

webserverSecretKeySecretName

The Secret name containing Flask secret_key for the Webserver.

~