Purpose:

To get out a useable KubernetesOperator and KubernetesExecutor we want to define what tasks should be included in the initial PRs of these products and what tasks should be reserved for later PRs

PRs for merging upstream:

To make the ultimate PR much more digestible, we will create a series of smaller PRs to minimize effort

• PR to allow executor_config
• PR for PodOperator with the pod building features basic kubernetes tests
• PR for k8s executor

Remaining Features

Kubernetes Operator:

• Create a Pod Operator
  • Pod Operator should be able to "return" a value for XCOM purposes 

Kubernetes Executor:

• Kubernetes health checker Unknown User (dimberman)
  • While the executor watches the pods that are spun up on kubernetes, we need to ensure the pods don't get "stuck" forever (for example, because a kubernetes secret you expected to exist does not exist)
    •  We chose a timeout based approach: 
      • allow ${x} seconds for the pod to start running, if that does not happen then kill the pod
    •  We do this by looking at a client-side timestamp. 
      • Every time `sync` is called we check the local data structure to see if any tasks have timed out, if they are we delete them
      • If the scheduler crashes we rehydrate this client-side timestamp by querying for all pending pods on kubernetes (restarting from 0)
      • While this technically leads to unbounded timeouts in the face of unbounded scheduler failures, the (pretty safe) assumption is scheduler failure is much less likely than pod failure
• Throttling + assurance of cluster health Unknown User (dimberman)
  • We don't want to overload a kubernetes cluster with too many pods/requesting too many resources, thus throttling airflow to prevent this would be nice

Up for debate:

Kubernetes Executor:

• Airflow configuration support (implementation):
  • Copy the airflow config into each worker pod as environmental variables 
    • (not safe from a security perspective, secrets in plaintext)
  • Force people to put `airflow.cfg` in a kubernetes secret (or persistent volume), mount that secret or persistent volume into each worker pod 
    • (nice but not the most flexible as it requires a very specific deployment setup)
  • On startup of the scheduler, snapshot the airflow configuration and create/update the airflow secret, then mount that secret into each worker pod 
    • (scheduler must have kubernetes RBAC rules to create/update 

Future Tasks:

Kubernetes Operator:

• Create a python custom resource so people can launch python jobs with arbitrary pip installs
  
• Possibly create a Job Operator