Purpose:

To get out a useable KubernetesOperator and KubernetesExecutor we want to define what tasks should be included in the initial PRs of these products and what tasks should be reserved for later PRs

...

PRs for merging upstream:

...

To make the ultimate PR much more digestible, we will create a series of smaller PRs to minimize effort

•  PR to allow executor_config
•  PR for PodOperator with the pod building features basic kubernetes tests
•  PR for k8s executor

Remaining Features

Kubernetes Operator:

•  Create a Pod Operator
  •  Pod Operator should be able to "return" a value for XCOM purposes 

Kubernetes Executor:

•  Executor storage + DAG transportation Executor should be able to transport large-scale DAG folders to slaves with minimal start-up 
  •  Git mode for development (init container on each worker uses git-sync pulls to pull dags from a git repo)
   Volume claim mode for production (use any `ReadOnlyMany` persistent volume claim to store and pull dags)
•  Airflow configuration support @benjigolberg PR - https://github.com/bloomberg/airflow/pull/9
  •  Changes made to airflow configuration on the scheduler should be reflected in every worker pod that is spun up. 
    •  There are a few ways to support this, see the "Up for debate" section
•  Task level configuration: Grant PR - https://github.com/bloomberg/airflow/pull/23 Should allow customizations of the kubernetes pods on a per-task basis. The minimum proposed options are:
  •  Resource usage customization
  •  Docker image customization
   Kubernetes secret customization + keychain (what does keychain refer to here)
  •  Service account customization
•  Crash safety of scheduler: Grant PR - https://github.com/bloomberg/airflow/pull/8
  •  The scheduler must tolerate unexpected crashes and start back up in a healthy state when restarting
    •  Use watcher to re-hydrate state from pods using labels instead of keeping data structures in memory
    •  Persist the watch resourceVersion and recover from crashes by starting the watch from the last applied resourceVersion 
    •  Clear queued tasks that haven't been launched on restart of the scheduler (by querying kubernetes)
•  Kubernetes health checker Unknown User (dimberman)
  •  While the executor watches the pods that are spun up on kubernetes, we need to ensure the pods don't get "stuck" forever (for example, because a kubernetes secret you expected to exist does not exist)
    •   We chose a timeout based approach: 
      •  allow ${x} seconds for the pod to start running, if that does not happen then kill the pod
    •   We do this by looking at a client-side timestamp. 
      •  Every time `sync` is called we check the local data structure to see if any tasks have timed out, if they are we delete them
      •  If the scheduler crashes we rehydrate this client-side timestamp by querying for all pending pods on kubernetes (restarting from 0)
      •  While this technically leads to unbounded timeouts in the face of unbounded scheduler failures, the (pretty safe) assumption is scheduler failure is much less likely than pod failure
•  Throttling + assurance of cluster health Unknown User (dimberman)
  •  We don't want to overload a kubernetes cluster with too many pods/requesting too many resources, thus throttling airflow to prevent this would be nice
    •  Resource-aware rate limiting is the gold standard, but will not be done in V1 (See the Future Tasks section)
    •  For v1, a simple "max pending pods" configuration value will stop airflow from launching any more pods until pods are moved out of the pending state (either because they timed out, succeeded, or died)
•  CI/testing environments Grant
  •  minikube cluster with `driver=None` mode on travis
  •  separate travis build executing kubernetes integration tests on minikube

Up for debate:

Kubernetes Executor:

•  Airflow configuration support (implementation):
  •  Copy the airflow config into each worker pod as environmental variables 
    •  (not safe from a security perspective, secrets in plaintext)
  •  Force people to put `airflow.cfg` in a kubernetes secret (or persistent volume), mount that secret or persistent volume into each worker pod 
    •  (nice but not the most flexible as it requires a very specific deployment setup)
  •  On startup of the scheduler, snapshot the airflow configuration and create/update the airflow secret, then mount that secret into each worker pod 
    •  (scheduler must have kubernetes RBAC rules to create/update 

Future Tasks:

Kubernetes Operator:

•  Create a python custom resource so people can launch python jobs with arbitrary pip installs
  
•  Possibly create a Job Operator
  

...