DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
...
Unlike the current MesosExecutor, which uses pickle to serialize DAGs and send them to pre-built slaves, the KubernetesExecutor will launch a new temporary worker job for each task. Each job will have contain a full airflow deployment and will run an airflow run <dag_id> <task_id> command.This design has two major benefits over the previous system. The first benefit is that dynamically creating airflow workers simplifies the cluster set-up. Users will not need to pre-build airflow workers or consider how the nodes will communicate with eachother. The second benefit is that dynamically creating pods allows for a highly elastic system that can easily scale to large workloads while not wasting resources during periods of low usage.
Monitoring jobs:
When we create the Kubernetes jobs, we will maintain a mapping of job_id -> job key. Using these job ids we can use the read_namespaced_jobs endpoint to consistently query kubernetes for the status of running jobs. Upon recieving a failure or success status from the API, the executor can forward the given state to the scheduler to show in the UI. By using the airflow batch job API (as opposed to launching pods), we get an assurance that any failed kubernetes job can retry a pre-set number of times before the executor kills the task.
1 2 3 | for job_id in current_jobs: status = api.read_namespaced_job(job_id, namespace).status process_status(job_id, key, status) |
...
We will watch jobs using the Kubernetes Watch API. This API will allow us to passively watch all events on a namespace, filtered by label. We can contain the watchers on separate threads which can use event handling to handle failures from airflow pods.
Sharing Dags:
To encourage a wide array of potential storage options for airflow users, we will take advantage of kubernetes persistent volume claims. With these claims, users will be allowed to use a wide variety of distributed storage options. We will have "airflow_importer" classes which can then be used to piggy-back persistentVolumeClaims to each task launched by the scheduler.
...