Page History

...

•  Airflow configuration support (implementation):
  •  Copy the airflow config into each worker pod as environmental variables 
    •  (not safe from a security perspective, secrets in plaintext)
  •  Force people to put `airflow.cfg` in a kubernetes secret (or persistent volume), mount that secret or persistent volume into each worker pod 
    •  (not very flexible)
  •  On startup of the scheduler, snapshot the airflow configuration and create/update the airflow secret, then mount that secret into each worker pod 
    •  (scheduler must have kubernetes RBAC rules to create/update secrets)
•  Kubernetes health checker (implementation):
  •  Timeout based: allow ${x} seconds for the pod to start running, if that does not happen then kill the pod
    •  Some choices here: server vs client side timeouts?
      •  server timeouts: requires periodic polling of the state of pending pods (separate from the watcher)
      •  client timeouts: requires no extra polling, but requires us to either:
        •  save the "launch" time of the taskinstance in the airflow database so we still have the launch time in the event of the scheduler crashing
        •  don't save the "launch" time, but if we crash then on startup the scheduler must kill all airflow pods that are not running (since we have no timeout information about them)
  •  State-machine based: look at the state of the pod during the watch and see if it is "unhealthy" in any way
    •  Requires deep kubernetes knowledge of every failure mode of a pod on kubernetes
•  Throttling + assurance of cluster health
  •  Two options to ensure we don't overload a kubernetes cluster with pods taking up too many resources:
    
    •  Do nothing, allow kubernetes to internally queue pods with the `Pending` status. (However, they will eventually be killed by the executor for timing out/taking too long to spin up)
      •  Pros: No work
      •  Cons: Shared kubernetes clusters can get overwhelmed. Can be worked around with namespace level resource quotas, airflow pools, or the airflow core `parallelism` option
    •  Actively try to rate limit jobs on a resource-aware level:
      •  Pros: can limit total airflow resource usage in a kubernetes cluster to ensure the cluster is healthy
      •  Cons: 
        •  what behavior do we want if the cluster is overworked? should we requeue tasks or fail them?
        •  rate limiting can become a slippery slope of extra features. if you statically say "airflow only allow allocating 100GB of memory to pods you launch" then the next thing you probably want is "airflow this group can use a total of 100GB of memory, this other group can use a total of 200GB of memory". then where do those "groups" come from (airflow users vs external integration with something like ldap)
•  Task level configuration (part 2)
  •  volume mount customization, labels, node selectors, tolerances, pod security contexts and capabilities (basically almost all of the kubernetes pod api could in theory be exposed)
•  Remove task heartbeating in kubernetes mode
  •  Since we have the watcher process that monitors the state of our pods, we technically don't need airflow's heartbeating of tasks. This heartbeating can be potentially expensive for the airflow metadata database at scale
    •  Can disable this heartbeating for as a performance optimization

...