Page History

airflowLocalSettings

airflow_local_settings file as a string (templated). You can bake an airflow_local_settings.py into your image instead. In that case, set this value to null.

See values.yaml

keep

airflowVersion

Airflow version (Used to make some decisions based on Airflow Version being deployed).

3.1.7

keep

apiSecretAnnotations

Annotations to add to the Api secret.

{}

Description remains from v2 terms. 

apiSecretKey

The Flask secret key for Airflow Api to encrypt browser session.

~

Description remains from v2 terms. 

apiServer.apiServerConfig

This string (templated) will be mounted into the Airflow API Server as a custom webserver_config.py. You can bake a webserver_config.py in to your image instead or specify a configmap containing the webserver_config.py.

~

Examples:

apiServerConfig: |-
  from airflow import configuration as conf

  # The SQLAlchemy connection string.
  SQLALCHEMY_DATABASE_URI = conf.get('database', 'SQL_ALCHEMY_CONN')

  # Flask-WTF flag for CSRF
  CSRF_ENABLED = True

This is still relevant if the user is using providers-fab and adding more fine-grained authentication integration through fab. That also generalised to BaseAuthManager and to providers that implement 3rd-party libraries and tools. I think the description should be updated.

webserver_config.py migration check mentioned above.

apiServer.apiServerConfigConfigMapName

The configmap name containing the webserver_config.py.

~

Examples:

apiServerConfigConfigMapName: my-api-server-configmap

keep

config

Settings to go into the mounted airflow.cfg

See values.yaml

keep

createUserJob.defaultUser.enabled

Enable default user creation.

True

keep

defaultAirflowDigest

Default airflow digest to deploy. Overrides tag.

~

defaultAirflowRepository

Default airflow repository. Overrides all the specific images below.

apache/airflow

keep

defaultAirflowTag

Default airflow tag to deploy.

3.1.7

keep

executor

Airflow executor.

CeleryExecutor

fernetKey

The Fernet key used to encrypt passwords (can only be set during install, not upgrade).

~

keep

fernetKeySecretAnnotations

Annotations to add to the Fernet Key secret.

{}

keep

jwtSecret

Secret key used to encode and decode JWTs to authenticate to public and private APIs (can only be set during install, not upgrade).

~

keep

jwtSecretAnnotations

Annotations to add to the JWT secret.

{}

keep

pgbouncer.enabled

Enable PgBouncer.

False

keep

pgbouncer.mountConfigSecret

Whether to mount the config secret files under /etc/pgbouncer/ by default.

True

keep

schedulerName

Specify kube scheduler name for Pods.

~

webserver.webserverConfig

This string (templated) will be mounted into the Airflow webserver as a custom webserver_config.py. You can bake a webserver_config.py in to your image instead or specify a configmap containing the webserver_config.py.

~

Examples:

webserverConfig: |-
  from airflow import configuration as conf

  # The SQLAlchemy connection string.
  SQLALCHEMY_DATABASE_URI = conf.get('database', 'SQL_ALCHEMY_CONN')

  # Flask-WTF flag for CSRF
  CSRF_ENABLED = True

webserver.webserverConfigConfigMapName

The configmap name containing the webserver_config.py.

~

Examples:

webserverConfigConfigMapName: my-webserver-configmap

webserverSecretAnnotations

Annotations to add to the webserver secret.

{}

webserverSecretKey

The Flask secret key for Airflow Webserver to encrypt browser session.

~

workers.schedulerName

Specify kube scheduler name for Airflow Celery workers objects and pods created with pod-template-file.

~

airflowHome

Airflow home directory. Used for mount paths.

/opt/airflow

keep

allowJobLaunching

Whether various Airflow components launch jobs.

False

It can be automated indeed. We can map to both a feature and a deprecation along with it. 

allowPodLaunching

Whether various Airflow components launch pods.

True

It can be automated indeed. We can map to both a feature and a deprecation along with it. 

apiSecretKeySecretName

The Secret name containing Flask secret_key for the Api.

~

Description remains from v2 terms. 

dags.gitSync.branch

Git branch

v2-2-stable

It makes sense. There are multiple ways in various systems of mounting the dags into the components. Making it bundled can help with easier additions on top of it for mounting operations.

dags.gitSync.containerName

Git sync container name.

git-sync

dags.gitSync.credentialsSecret

Name of a Secret containing the repo GIT_SYNC_USERNAME and GIT_SYNC_PASSWORD.

~

dags.gitSync.depth

Repository depth.

1

dags.gitSync.emptyDirConfig

Configuration for dags empty dir volume.

~

dags.gitSync.enabled

Enable Git sync.

False

dags.gitSync.env

Environment variables for git sync container.

[]

Examples:

env:
- name: GIT_SYNC_TIMEOUT
  value: '60'
env:
- name: GIT_SYNC_USERNAME
  valueFrom:
    secretKeyRef:
      key: username
      name: git-secret

dags.gitSync.envFrom

Extra envFrom ‘items’ that will be added to the definition of Airflow gitSync containers; a string or array are expected (templated).

~

Examples:

envFrom: |-
  - secretRef:
      name: 'proxy-config
envFrom: |-
  - configMapRef:
      name: 'proxy-config

dags.gitSync.extraVolumeMounts

Mount additional volumes into git sync container.

[]

dags.gitSync.knownHosts

When using a ssh private key, the contents of your known_hosts file.

~

Examples:

knownHosts: |-
  <host1>,<ip1> <key1>
  <host2>,<ip2> <key2>
knownHosts: <host1>,<ip1> <key1>

dags.gitSync.maxFailures

The number of consecutive failures allowed before aborting.

0

dags.gitSync.period

Interval between git sync attempts in Go-style duration string. High values are more likely to cause DAGs to become out of sync between different components. Low values cause more traffic to the remote git repository.

5s

dags.gitSync.ref

Git revision branch, tag, or hash.

v2-2-stable

dags.gitSync.repo

Git repository.

https://github.com/apache/airflow.git

dags.gitSync.resources

Resources on workers git-sync sidecar

{}

Examples:

resources:
  limits:
    cpu: 100m
    memory: 128Mi
  requests:
    cpu: 100m
    memory: 128Mi

dags.gitSync.rev

Git revision.

HEAD

dags.gitSync.securityContext

Security context for the gitSync container (deprecated, use securityContexts instead). If not set, the values from securityContext will be used.

{}

Examples:

securityContext:
  runAsGroup: 0
  runAsUser: 50000

dags.gitSync.sshKey

SSH private key

~

dags.gitSync.sshKeySecret

Name of a Secret containing the repo sshKeySecret.

~

dags.gitSync.subPath

Subpath within the repo where dags are located.

tests/dags

dags.gitSync.uid

Git sync container run as user parameter.

65533

dags.gitSync.wait

Interval between git sync attempts in seconds. High values are more likely to cause DAGs to become out of sync between different components. Low values cause more traffic to the remote git repository.

~

dags.mountPath

Where dags volume will be mounted. Works for both persistence and gitSync. If not specified, dags mount path will be set to $AIRFLOW_HOME/dags

~

dags.persistence.accessMode

Access mode of the persistent volume.

ReadWriteOnce

  keep

dags.persistence.annotations

Annotations for the dag PVC

{}

  keep

dags.persistence.enabled

Enable persistent volume for storing dags.

False

  keep

dags.persistence.existingClaim

The name of an existing PVC to use.

~

  keep

dags.persistence.size

Volume size for dags.

1Gi

  keep

dags.persistence.storageClassName

If using a custom StorageClass, pass name here (templated).

~

  keep

dags.persistence.subPath

Subpath within the PVC where dags are located.

~

  keep

elasticsearch.connection

Elasticsearch connection configuration.

{}

Examples:

connection:
  host: '...'
  pass: '...'
  port: '...'
  scheme: https
  user: '...'

elasticsearch.connection.host

Host

""

elasticsearch.connection.pass

Password

""

elasticsearch.connection.port

Port

80

elasticsearch.connection.scheme

Scheme

http

elasticsearch.connection.user

Username

""

elasticsearch.enabled

Enable Elasticsearch task logging.

False

elasticsearch.secretAnnotations

Extra annotations to apply to the elasticsearch secret.

{}

elasticsearch.secretName

A secret containing the connection string.

~

enableBuiltInSecretEnvVars.AIRFLOW_CONN_AIRFLOW_DB

Enable AIRFLOW_CONN_AIRFLOW_DB variable to be read from the Metadata Secret

True

SQLAlchemy connection string from K8s secret. Keeping with a better description and maybe renaming to reflect better.

enableBuiltInSecretEnvVars.AIRFLOW__API_AUTH__JWT_SECRET

Enable AIRFLOW__API_AUTH__JWT_SECRET variable to be read from the JWT Secret

True

Enable using K8S secret in env var to pass jwt_secret config keep

enableBuiltInSecretEnvVars.AIRFLOW__API__SECRET_KEY

Enable AIRFLOW__API__SECRET_KEY variable to be read from the Api Secret Key Secret

True

Similar to above

secret_key keep

enableBuiltInSecretEnvVars.AIRFLOW__CELERY__BROKER_URL

Enable AIRFLOW__CELERY__BROKER_URL variable to be read from the Celery Broker URL Secret

True

All these are the same as above, as setting some config that enables to read secret to pass to the env var. Maybe move them to Kustomize?

enableBuiltInSecretEnvVars.AIRFLOW__CELERY__CELERY_RESULT_BACKEND

Enable AIRFLOW__CELERY__CELERY_RESULT_BACKEND variable to be read from the Celery Result Backend Secret - Airflow 1.10.* variant

True

enableBuiltInSecretEnvVars.AIRFLOW__CELERY__RESULT_BACKEND

Enable AIRFLOW__CELERY__RESULT_BACKEND variable to be read from the Celery Result Backend Secret

True

enableBuiltInSecretEnvVars.AIRFLOW__CORE__FERNET_KEY

Enable AIRFLOW__CORE__FERNET_KEY variable to be read from the Fernet key Secret

True

enableBuiltInSecretEnvVars.AIRFLOW__CORE__SQL_ALCHEMY_CONN

Enable AIRFLOW__CORE__SQL_ALCHEMY_CONN variable to be read from the Metadata Secret

True

enableBuiltInSecretEnvVars.AIRFLOW__DATABASE__SQL_ALCHEMY_CONN

Enable AIRFLOW__DATABASE__SQL_ALCHEMY_CONN variable to be read from the Metadata Secret

True

enableBuiltInSecretEnvVars.AIRFLOW__ELASTICSEARCH__ELASTICSEARCH_HOST

Enable AIRFLOW__ELASTICSEARCH__ELASTICSEARCH_HOST variable to be read from the Elasticsearch Host Secret - Airflow <1.10.4 variant

True

enableBuiltInSecretEnvVars.AIRFLOW__ELASTICSEARCH__HOST

Enable AIRFLOW__ELASTICSEARCH__HOST variable to be read from the Elasticsearch Host Secret

True

enableBuiltInSecretEnvVars.AIRFLOW__OPENSEARCH__HOST

Enable AIRFLOW__OPENSEARCH__HOST variable to be read from the OpenSearch Host Secret

True

enableBuiltInSecretEnvVars.AIRFLOW__WEBSERVER__SECRET_KEY

Enable AIRFLOW__WEBSERVER__SECRET_KEY variable to be read from the Webserver Secret Key Secret

True

env

Environment variables for all Airflow containers.

[]

Examples:

env:
- name: MYENVVAR
  value: something_fun

extraEnv

Extra env ‘items’ that will be added to the definition of Airflow containers; a string is expected (templated).

~

Examples:

extraEnv: |-
  - name: AIRFLOW__CORE__LOAD_EXAMPLES
     value: True

extraEnvFrom

Extra envFrom ‘items’ that will be added to the definition of Airflow containers; a string is expected (templated).

~

Examples:

extraEnvFrom: |-
  - secretRef:
      name: '{{ .Release.Name }}-airflow-connections'
extraEnvFrom: |-
  - configMapRef:
      name: '{{ .Release.Name }}-airflow-variables'

fernetKeySecretName

The Fernet key secret name.

~

gid

Group of airflow user.

0

jwtSecretName

The JWT secret name.

~

logs.emptyDirConfig

Configuration for logs empty dir volume.

~

  keep

logs.persistence.annotations

Annotations to add to logs PVC

{}

  keep

logs.persistence.enabled

Enable persistent volume for storing logs.

False

  keep

logs.persistence.existingClaim

The name of an existing PVC to use.

~

  keep

logs.persistence.size

Volume size for logs.

100Gi

  keep

logs.persistence.storageClassName

If using a custom StorageClass, pass name here (templated).

~

  keep

logs.persistence.subPath

The subpath of the existing PVC to use.

~

  keep

multiNamespaceMode

Whether Airflow can launch workers and/or pods in multiple namespaces. If true, it creates ClusterRole/ClusterRolebinding (with access to entire cluster)

False 

opensearch.connection

OpenSearch connection configuration.

{}

Examples:

connection:
  host: '...'
  pass: '...'
  port: '...'
  scheme: https
  user: '...'

opensearch.connection.host

Host

""

opensearch.connection.pass

Password

""

opensearch.connection.port

Port

80

opensearch.connection.scheme

Scheme

http

opensearch.connection.user

Username

""

opensearch.enabled

Enable OpenSearch task logging.

False

opensearch.secretName

A secret containing the connection string.

~

podTemplate

The content of pod_template_file.yaml used for KubernetesExecutor workers (templated). The default (see files/pod-template-file.kubernetes-helm-yaml) already takes into account normal workers configuration parameters (e.g. workers.resources), so you normally won’t need to override this directly.

~

Examples:

podTemplate: |-
  apiVersion: v1
  kind: Pod
  metadata:
    name: placeholder-name
    labels:
      tier: airflow
      component: worker
      release: {{ .Release.Name }}
  spec:
    priorityClassName: high-priority
    containers:
      - name: base
      ...

secret

Secrets for all Airflow containers.

[]

Examples:

secret:
- envName: SecretEnvVar
  secretKey: somekey
  secretName: somesecret

uid

User of airflow user.

50000

of airflow user.

50000

volumeMounts

VolumeMounts for all Airflow containers.

[]

volumes

Volumes for all Airflow containers.

[]

webserverSecretKeySecretName

The Secret name containing Flask secret_key for the Webserver.

~

apiServer.affinity

Specify scheduling constraints for API server pods.

See values.yaml

apiServer.allowPodLogReading

Allow API server to read k8s pod logs. Useful when you don’t have an external log store.

True

apiServer.annotations

Annotations to add to the API server deployment

{}

apiServer.args

Args to use when running the Airflow API server (templated). When running behind a reverse proxy, add –proxy-headers to enable Uvicorn to respect X-Forwarded-Proto, X-Forwarded-For, and X-Forwarded-Port headers.

['bash', '-c', 'exec airflow api-server']

Examples:

args:
- bash
- -c
- exec airflow api-server --proxy-headers

apiServer.command

Command to use when running the Airflow API server (templated).

~

apiServer.configMapAnnotations

Extra annotations to apply to the API server configmap.

{}

apiServer.enabled

Enable Airflow API server deployment.

True

apiServer.env

Add additional env vars to API server. When running behind a reverse proxy, set FORWARDED_ALLOW_IPS to specify which IPs are trusted to send X-Forwarded-* headers. Use “*” for trusted environments, or specify proxy IP ranges for production.

[]

Examples:

env:
- - name: FORWARDED_ALLOW_IPS
    value: '*'

apiServer.extraContainers

Launch additional containers into API server.

[]

apiServer.extraInitContainers

Add additional init containers into API server.

[]

apiServer.extraVolumeMounts

Mount additional volumes into API server.

[]

apiServer.extraVolumes

Mount additional volumes into API server.

[]

apiServer.hostAliases

HostAliases for the API server pod.

[]

Examples:

hostAliases:
- hostnames:
  - foo.local
  ip: 127.0.0.1
hostAliases:
- hostnames:
  - foo.remote
  ip: 10.1.2.3

apiServer.hpa.behavior

HorizontalPodAutoscalerBehavior configures the scaling behavior of the target.

{}

apiServer.hpa.enabled

Enable HPA autoscaling for API server

False

apiServer.hpa.maxReplicaCount

Maximum number of API server replicas created by HPA if HPA is enabled.

5

apiServer.hpa.metrics

Specifications for which to use to calculate the desired replica count.

[{'type': 'Resource', 'resource': {'name': 'cpu', 'target': {'type': 'Utilization', 'averageUtilization': 50}}}] 

apiServer.hpa.minReplicaCount

Minimum number of API server replicas created by HPA if HPA is enabled.

1

apiServer.labels

Labels to add to the API server objects and pods.

{}

apiServer.livenessProbe.failureThreshold

API server Liveness probe failure threshold.

5

apiServer.livenessProbe.initialDelaySeconds

API server Liveness probe initial delay.

15

apiServer.livenessProbe.periodSeconds

API server Liveness probe period seconds.

10

apiServer.livenessProbe.scheme

API server Liveness probe scheme.

HTTP

apiServer.livenessProbe.timeoutSeconds

API server Liveness probe timeout seconds.

5

apiServer.networkPolicy.ingress.from

Peers for API server NetworkPolicyingress.

[]

apiServer.networkPolicy.ingress.ports

Ports for API server NetworkPolicyingress (if from is set).

[{'port': '{{ .Values.ports.apiServer }}'}]

Examples:

ports:
- port: 8080

apiServer.nodeSelector

Select certain nodes for API server pods.

{}

apiServer.podAnnotations

Annotations to add to the API server pods.

{}

apiServer.podDisruptionBudget.config.maxUnavailable

Max unavailable pods for API server.

1

apiServer.podDisruptionBudget.config.minAvailable

Min available pods for API server.

1 

apiServer.podDisruptionBudget.enabled

Enable pod disruption budget.

False

apiServer.priorityClassName

Specify priority for API server pods.

~

apiServer.readinessProbe.failureThreshold

API server Readiness probe failure threshold.

5

apiServer.readinessProbe.initialDelaySeconds

API server Readiness probe initial delay.

15

apiServer.readinessProbe.periodSeconds

API server Readiness probe period seconds.

10

apiServer.readinessProbe.scheme

API server Readiness probe scheme.

HTTP

apiServer.readinessProbe.timeoutSeconds

API server Readiness probe timeout seconds.

5

apiServer.replicas

How many Airflow API server replicas should run. This setting is ignored when HPA (Horizontal Pod Autoscaler) is enabled

1

apiServer.resources

Resources for API server pods.

{}

Examples:

resources:
  limits:
    cpu: 100m
    memory: 128Mi
  requests:
    cpu: 100m
    memory: 128Mi

apiServer.service.annotations

Annotations for the API server Service.

{}

apiServer.service.loadBalancerIP

API server Service loadBalancerIP.

~

apiServer.service.loadBalancerSourceRanges

API server Service loadBalancerSourceRanges.

[]

Examples:

loadBalancerSourceRanges:
- 10.123.0.0/16

apiServer.service.ports

Ports for the API server Service.

[{'name': 'api-server', 'port': '{{ .Values.ports.apiServer }}'}]

Examples:

ports:
- name: api-server
  port: 8080
  targetPort: api-server
ports:
- name: only_sidecar
  port: 9080
  targetPort: 8888

apiServer.service.type

API server Service type.

ClusterIP

apiServer.serviceAccount.annotations

Annotations to add to the API server Kubernetes ServiceAccount.

{}

apiServer.serviceAccount.automountServiceAccountToken

Specifies if ServiceAccount’s API credentials should be mounted onto Pods.

True

apiServer.serviceAccount.create

Specifies whether a ServiceAccount should be created.

True

apiServer.serviceAccount.name

The name of the ServiceAccount to use. If not set and create is true, a name is generated using the release name.

~

apiServer.startupProbe.failureThreshold

API server Startup probe failure threshold.

6

apiServer.startupProbe.initialDelaySeconds

API server Startup probe initial delay seconds.

0

apiServer.startupProbe.periodSeconds

API server Startup probe period seconds.

10

apiServer.startupProbe.scheme

API server Startup probe scheme.

HTTP

apiServer.startupProbe.timeoutSeconds

API server Startup probe timeout seconds.

20

apiServer.strategy

Specifies the strategy used to replace old Pods by new ones.

~

apiServer.tolerations

Specify Tolerations for API server pods.

[]

apiServer.waitForMigrations.enabled

Enable wait-for-airflow-migrations init container.

True

apiServer.waitForMigrations.env

Add additional env vars to wait-for-airflow-migrations init container.

[]