Purpose:

To get out a useable KubernetesOperator and KubernetesExecutor we want to define what tasks should be included in the initial PRs of these products and what tasks should be reserved for later PRs

...

PRs for merging upstream:

...

To make the ultimate PR much more digestible, we will create a series of smaller PRs to minimize effort

•  PR to allow executor_config
•  PR for PodOperator with the pod building features basic kubernetes tests
•  PR for k8s executor

Remaining Features

Kubernetes Operator:

•  Create a Pod Operator
  •  Pod Operator should be able to "return" a value for XCOM purposes 

Kubernetes Executor:

•  Executor storage + DAG transportation
  •  Executor should be able to transport large-scale DAG folders to slaves with minimal start-up 
    •  Git mode for development (init container on each worker uses git-sync pulls to pull dags from a git repo)
    •  Volume claim mode for production (use any `ReadOnlyMany` persistent volume claim to store and pull dags)
•  Airflow configuration support
  •  Changes made to airflow configuration on the scheduler should be reflected in every worker pod that is spun up. 
    •  There are a few ways to support this, see the "Up for debate" section
•  Task level configuration: 
  •  Should allow customizations of the kubernetes pods on a per-task basis. The minimum proposed options are:
    •  Resource usage customization
    •  Docker image customization
    •  Kubernetes secret customization + keychain (what does keychain refer to here)
    •  Service accounts
•  Crash safety of scheduler
  •  The scheduler must tolerate unexpected crashes and start back up in a healthy state when restarting
    •  Use watcher to re-hydrate state from pods using labels instead of keeping data structures in memory
    •  Persist the watch resourceVersion and recover from crashes by starting the watch from the last applied resourceVersion 
    •  Clear all queued tasks on restart of the scheduler
•  Kubernetes health checker  Kubernetes health checker Unknown User (dimberman)
  •  While the executor watches the pods that are spun up on kubernetes, we need to ensure the pods don't get "stuck" forever (for example, because a kubernetes secret you expected to exist does not exist)
    •   There are a few ways to support this, see the "Up for debate" section
•  CI/testing environments
  •  minikube cluster with `driver=None` mode on travis
  •  separate travis build executing kubernetes integration tests on minikube
• • •   We chose a timeout based approach: 
      •  allow ${x} seconds for the pod to start running, if that does not happen then kill the pod
    •   We do this by looking at a client-side timestamp. 
      •  Every time `sync` is called we check the local data structure to see if any tasks have timed out, if they are we delete them
      •  If the scheduler crashes we rehydrate this client-side timestamp by querying for all pending pods on kubernetes (restarting from 0)
      •  While this technically leads to unbounded timeouts in the face of unbounded scheduler failures, the (pretty safe) assumption is scheduler failure is much less likely than pod failure
•  Throttling + assurance of cluster health Unknown User (dimberman)
  •  We don't want to overload a kubernetes cluster with too many pods/requesting too many resources, thus throttling airflow to prevent this would be nice

Up for debate:

Kubernetes Executor:

•  Airflow configuration support (implementation):
  •  Copy the airflow config into each worker pod as environmental variables 
    •  (not safe from a security perspective, secrets in plaintext)
  •  Force people to put `airflow.cfg` in a kubernetes secret (or persistent volume), mount that secret or persistent volume into each worker pod 
    •  (nice but not very flexiblethe most flexible as it requires a very specific deployment setup)
  •  On startup of the scheduler, snapshot the airflow configuration and create/update the airflow secret, then mount that secret into each worker pod 
    •  (scheduler must have kubernetes RBAC rules to create/update secrets)
•  Kubernetes health checker (implementation):
  •  Timeout based: allow ${x} seconds for the pod to start running, if that does not happen then kill the pod
    •  Some choices here: server vs client side timeouts?
      •  server timeouts: requires periodic polling of the state of pending pods (separate from the watcher)
      •  client timeouts: requires no extra polling, but requires us to either:
        •  save the "launch" time of the taskinstance in the airflow database so we still have the launch time in the event of the scheduler crashing
        •  don't save the "launch" time, but if we crash then on startup the scheduler must kill all airflow pods that are not running (since we have no timeout information about them)
  •  State-machine based: look at the state of the pod during the watch and see if it is "unhealthy" in any way
    •  Requires deep kubernetes knowledge of every failure mode of a pod on kubernetes
 Throttling + assurance of cluster health Two options to ensure we don't overload a kubernetes cluster with pods taking up too many resources:

•  Do nothing, allow kubernetes to internally queue pods with the `Pending` status. (However, they will eventually be killed by the executor for timing out/taking too long to spin up)
  •  Pros: No work
  •  Cons: Shared kubernetes clusters can get overwhelmed. Can be worked around with namespace level resource quotas, airflow pools, or the airflow core `parallelism` option
 
• • • update 

Future Tasks:

Kubernetes Operator:

•  Create a python custom resource so people can launch python jobs with arbitrary pip installs
  
•  Possibly create a Job Operator
  

...

...

...

...

Future Tasks:

Kubernetes Operator:

•  Create a python custom resource so people can launch python jobs with arbitrary pip installs
  
•  Possibly create a Job Operator
  

...