Purpose:

To get out a useable KubernetesOperator and KubernetesExecutor we want to define what tasks should be included in the initial PRs of these products and what tasks should be reserved for later PRs

Initial PR:

These are the features that we would consider "crucial infrastructure." While these features will not allow the full flexibility desired by kubernetes proficient users, it will allow small and large-scale airflow instances to run natively on kubernetes.

Kubernetes Operator:

• Create a Pod Operator
  • Pod Operator should be able to "return" a value for XCOM purposes 

Kubernetes Executor:

• Executor storage + DAG transportation
  • Executor should be able to transport large-scale DAG folders to slaves with minimal start-up 
    • Git mode for development (init container on each worker uses git-sync pulls to pull dags from a git repo)
    • Volume claim mode for production (use any `ReadOnlyMany` persistent volume claim to store and pull dags)
• Airflow configuration support
  • Changes made to airflow configuration on the scheduler should be reflected in every worker pod that is spun up. 
    • There are a few ways to support this, see the "Up for debate" section
• Task level configuration: 
  • Should allow customizations of the kubernetes pods on a per-task basis. The minimum proposed options are:
    • Resource usage customization
    • Docker image customization
    • Kubernetes secret customization + keychain (what does keychain refer to here)
    • Service account customization
• Crash safety of scheduler
  • The scheduler must tolerate unexpected crashes and start back up in a healthy state when restarting
    • Use watcher to re-hydrate state from pods using labels instead of keeping data structures in memory
    • Persist the watch resourceVersion and recover from crashes by starting the watch from the last applied resourceVersion 
    • Clear all queued tasks on restart of the scheduler
• Kubernetes health checker
  • While the executor watches the pods that are spun up on kubernetes, we need to ensure the pods don't get "stuck" forever (for example, because a kubernetes secret you expected to exist does not exist)
    •  We chose a timeout based approach: 
      • allow ${x} seconds for the pod to start running, if that does not happen then kill the pod
    •  We do this by looking at a client-side timestamp. 
      • Every time `sync` is called we check the local data structure to see if any tasks have timed out, if they are we delete them
      • If the scheduler crashes we rehydrate this client-side timestamp by querying for all pending pods on kubernetes (restarting from 0)
      • While this technically leads to unbounded timeouts in the face of unbounded scheduler failures, the (pretty safe) assumption is scheduler failure is much less likely than pod failure
• CI/testing environments
  • minikube cluster with `driver=None` mode on travis
  • separate travis build executing kubernetes integration tests on minikube

Up for debate:

Kubernetes Executor:

• Airflow configuration support (implementation):
  • Copy the airflow config into each worker pod as environmental variables 
    • (not safe from a security perspective, secrets in plaintext)
  • Force people to put `airflow.cfg` in a kubernetes secret (or persistent volume), mount that secret or persistent volume into each worker pod 
    • (not very flexible)
  • On startup of the scheduler, snapshot the airflow configuration and create/update the airflow secret, then mount that secret into each worker pod 
    • (scheduler must have kubernetes RBAC rules to create/update secrets)
• Throttling + assurance of cluster health
  • Two options to ensure we don't overload a kubernetes cluster with pods taking up too many resources:
    
    • Do nothing, allow kubernetes to internally queue pods with the `Pending` status. (However, they will eventually be killed by the executor for timing out/taking too long to spin up)
      • Pros: No work
      • Cons: Shared kubernetes clusters can get overwhelmed. Can be worked around with namespace level resource quotas, airflow pools, or the airflow core `parallelism` option
    • Actively try to rate limit jobs on a resource-aware level:
      • Pros: can limit total airflow resource usage in a kubernetes cluster to ensure the cluster is healthy
      • Cons: 
        • what behavior do we want if the cluster is overworked? should we requeue tasks or fail them?
        • rate limiting can become a slippery slope of extra features. if you statically say "airflow only allow allocating 100GB of memory to pods you launch" then the next thing you probably want is "airflow this group can use a total of 100GB of memory, this other group can use a total of 200GB of memory". then where do those "groups" come from (airflow users vs external integration with something like ldap)

Future Tasks:

Kubernetes Operator:

• Create a python custom resource so people can launch python jobs with arbitrary pip installs
  
• Possibly create a Job Operator