DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
...
- Executor storage + DAG transportation
- Executor should be able to transport large-scale DAG folders to slaves with minimal start-up
- Git mode for development (init container on each worker uses git-sync pulls to pull dags from a git repo)
- Volume claim mode for production (use any `ReadOnlyMany` persistent volume claim to store and pull dags)
- Executor should be able to transport large-scale DAG folders to slaves with minimal start-up
- Airflow configuration supportsupport @benjigolberg PR - https://github.com/bloomberg/airflow/pull/9
- Changes made to airflow configuration on the scheduler should be reflected in every worker pod that is spun up.
- There are a few ways to support this, see the "Up for debate" section
- Changes made to airflow configuration on the scheduler should be reflected in every worker pod that is spun up.
- Task level configuration: @benjigolberg PR Grant PR - https://github.com/bloomberg/airflow/pull/923
- Should allow customizations of the kubernetes pods on a per-task basis. The minimum proposed options are:
- Resource usage customization
- Docker image customization
- Kubernetes secret customization + keychain (what does keychain refer to here)
- Service account customization
- Should allow customizations of the kubernetes pods on a per-task basis. The minimum proposed options are:
- Crash safety of scheduler: Grant PR - https://github.com/bloomberg/airflow/pull/8
- The scheduler must tolerate unexpected crashes and start back up in a healthy state when restarting
- Use watcher to re-hydrate state from pods using labels instead of keeping data structures in memory
- Persist the watch resourceVersion and recover from crashes by starting the watch from the last applied resourceVersion
- Clear queued tasks that haven't been launched on restart of the scheduler (by querying kubernetes)
- The scheduler must tolerate unexpected crashes and start back up in a healthy state when restarting
- Kubernetes health checker Unknown User (dimberman)
- While the executor watches the pods that are spun up on kubernetes, we need to ensure the pods don't get "stuck" forever (for example, because a kubernetes secret you expected to exist does not exist)
- We chose a timeout based approach:
- allow ${x} seconds for the pod to start running, if that does not happen then kill the pod
- We do this by looking at a client-side timestamp.
- Every time `sync` is called we check the local data structure to see if any tasks have timed out, if they are we delete them
- If the scheduler crashes we rehydrate this client-side timestamp by querying for all pending pods on kubernetes (restarting from 0)
- While this technically leads to unbounded timeouts in the face of unbounded scheduler failures, the (pretty safe) assumption is scheduler failure is much less likely than pod failure
- We chose a timeout based approach:
- While the executor watches the pods that are spun up on kubernetes, we need to ensure the pods don't get "stuck" forever (for example, because a kubernetes secret you expected to exist does not exist)
- Throttling + assurance of cluster health Unknown User (dimberman)
- We don't want to overload a kubernetes cluster with too many pods/requesting too many resources, thus throttling airflow to prevent this would be nice
- Resource-aware rate limiting is the gold standard, but will not be done in V1 (See the Future Tasks section)
- For v1, a simple "max pending pods" configuration value will stop airflow from launching any more pods until pods are moved out of the pending state (either because they timed out, succeeded, or died)
- We don't want to overload a kubernetes cluster with too many pods/requesting too many resources, thus throttling airflow to prevent this would be nice
- CI/testing environments Grant
- minikube cluster with `driver=None` mode on travis
- separate travis build executing kubernetes integration tests on minikube
...