Page History

Kubernetes Executor:

•  Throttling + assurance of cluster health
  •  Actively try to rate limit jobs on a resource-aware level:
    •  Can set limits like "only allow airflow to have pods in flight with a sum total resource request of 400GB of memory and 100CPUs or less"
      •  This can be done by querying kubernetes with the label selector "airflow-slave" and adding all up the used cpus/memory
    •  Even further in the future this could be extended to allow dynamic configuration of the memory/cpu resource limits
      •  Imagine "group A" can use a total of 100GB of memory and 20CPUs but "group B" can use a total of 200GB of memory and 40CPUs
•  Task level configuration (part 2)configuration 
  •  volume mount customization, labels, node selectors, tolerances, pod security contexts and capabilities (basically almost all of the kubernetes pod api could in theory be exposed)
•  Remove task heartbeating in kubernetes mode
  •  Since we have the watcher process that monitors the state of our pods, we technically don't need airflow's heartbeating of tasks. This heartbeating can be potentially expensive for the airflow metadata database at scale
    •  Can disable this heartbeating as a performance optimization
•  Airflow file-level logging
  •  Not hard to implement in code, but requires mounting a `ReadWriteMany` persistent volume for logging into each worker (RWM volume is a heavy dependency)
  •  Is this needed when there are options like S3 for logging?
•  We can create a pending queue that the watcher can remove from when it receives a running event (does this still apply?)