[WIP] AIP-66: Execution of specific DAG versions

Status

This AIP is part of AIP-63, which aims to add DAG Versioning to Airflow.

This AIP supersedes AIP-5 and AIP-20 (when I update this AIP to include a manifest concept - coming soon!).

This AIP is very much still a work in progress. It is the next (and likely final) step in the path to supporting DAG Versioning in Airflow. All of this is subject to change, but shows the high level vision of what this feature will allow.

Motivation

Today Airflow will always execute a task using the latest DAG code. While in some cases this may be what users want, in others it may be desired to complete a whole DAG run on a single version of a DAGs code. Or, when clearing tasks in older runs, allow that task to run on the DAG code at the time of the original run.

Considerations

We will introduce the concept of a DAG bundle (a collection of DAG files and other files) and allow Airflow to choose which version of a DAG bundle to use for a given task try. This means that a DAG run could continue running on the same DAG code for the entire run, even if the DAG is changed mid way through, as the worker can retrieve a specific DAG bundle when running a task.

This will require Airflow to support a different way of finding DAG bundles - it can no longer simply expect them on local disk. This will be done in a pluggable way by building in support for DAG bundle backends, with optional versioning support, so the ecosystem can evolve as time goes on.

By default DAG runs will execute on the same DAG bundle for the whole run, assuming the DAG bundle backend supports versioning. However, DAGs can opt to continue the existing behavior of running on the latest DAG code instead.

Pluggable DAG bundle backends

DAG bundle backends will allow Airflow to retrieve DAG bundles at a given version at any point in time. Since it’s common that DAGs are not simply contained in a single file, we require a way to version a whole set of files at the same time. How this is accomplished will depend on the backend. Airflow will only operate on complete DAG bundles - it will never attempt to identify or fetch a subset of a DAG bundle.

While we are still determining which DAG bundle backends will be available initially, it is likely that we have support for git and versioned buckets (like s3) from day 1.

It will be possible to have a DAG bundle backend that does not support versioning as well. This is primarily to allow a native local dev experience, but does open the door to other remote backends that may not meet Airflow versioning requirements.

By default Airflow will configure a local filesystem backed DAG bundle backend that does not support versioning, to support the local dev experience. This backend will expose DAGs from the currently configured DAG directory to Airflow, maintaining backward compatibility.

Parsing and the scheduler

Parsing will always happen on the latest version of the DAG bundles, as is the case today. The scheduler will also use the latest version when dealing with new DAG runs.

The scheduler will also be able to schedule and send to the executor a task that runs on a prior version of a DAG bundle.

Worker

The executor will now tell the worker what DAG bundle a task needs to run against. The worker can then use the DAG bundle backend to get the dags and execute the task.