DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
...
- Airflow configuration support (implementation):
- Copy the airflow config into each worker pod as environmental variables
- (not safe from a security perspective, secrets in plaintext)
- Force people to put `airflow.cfg` in a kubernetes secret (or persistent volume), mount that secret or persistent volume into each worker pod
- (not very flexible)
- On startup of the scheduler, snapshot the airflow configuration and create/update the airflow secret, then mount that secret into each worker pod
- (scheduler must have kubernetes RBAC rules to create/update secrets)
- Copy the airflow config into each worker pod as environmental variables
- Throttling + assurance of cluster health
- Two options to ensure we don't overload a kubernetes cluster with pods taking up too many resources:
- Do nothing, allow kubernetes to internally queue pods with the `Pending` status. (However, they will eventually be killed by the executor for timing out/taking too long to spin up)
- Pros: No work
- Cons: Shared kubernetes clusters can get overwhelmed. Can be worked around with namespace level resource quotas, airflow pools, or the airflow core `parallelism` option
- Actively try to rate limit jobs on a resource-aware level:
- Pros: can limit total airflow resource usage in a kubernetes cluster to ensure the cluster is healthy
- Cons:
- what behavior do we want if the cluster is overworked? should we requeue tasks or fail them?
- rate limiting can become a slippery slope of extra features. if you statically say "airflow only allow allocating 100GB of memory to pods you launch" then the next thing you probably want is "airflow this group can use a total of 100GB of memory, this other group can use a total of 200GB of memory". then where do those "groups" come from (airflow users vs external integration with something like ldap)
- Do nothing, allow kubernetes to internally queue pods with the `Pending` status. (However, they will eventually be killed by the executor for timing out/taking too long to spin up)
- Two options to ensure we don't overload a kubernetes cluster with pods taking up too many resources:
- update
Future Tasks:
Kubernetes Operator:
...
Kubernetes Executor:
- Throttling + assurance of cluster health
- Actively try to rate limit jobs on a resource-aware level:
- Can set limits like "only allow airflow to have pods in flight with a sum total resource request of 400GB of memory and 100CPUs or less"
- This can be done by querying kubernetes with the label selector "airflow-slave" and adding all up the used cpus/memory
- Even further in the future this could be extended to allow dynamic configuration of the memory/cpu resource limits
- Imagine "group A" can use a total of 100GB of memory and 20CPUs but "group B" can use a total of 200GB of memory and 40CPUs
- Can set limits like "only allow airflow to have pods in flight with a sum total resource request of 400GB of memory and 100CPUs or less"
- Actively try to rate limit jobs on a resource-aware level:
- Task level configuration (part 2)
- volume mount customization, labels, node selectors, tolerances, pod security contexts and capabilities (basically almost all of the kubernetes pod api could in theory be exposed)
- Remove task heartbeating in kubernetes mode
- Since we have the watcher process that monitors the state of our pods, we technically don't need airflow's heartbeating of tasks. This heartbeating can be potentially expensive for the airflow metadata database at scale
- Can disable this heartbeating as a performance optimization
- Since we have the watcher process that monitors the state of our pods, we technically don't need airflow's heartbeating of tasks. This heartbeating can be potentially expensive for the airflow metadata database at scale
- Airflow file-level logging
- Not hard to implement in code, but requires mounting a `ReadWriteMany` persistent volume for logging into each worker (RWM volume is a heavy dependency)
- Is this needed when there are options like S3 for logging?
- We can create a pending queue that the watcher can remove from when it receives a running event (does this still apply?)