Page History

Status

Motivation

Airflow 3 introduces DAG Versioning (AIP-63: DAG Versioning), allowing the system to track the historical state of DAGs. However, the scheduler continues to automatically use the latest parsed DagVersion for all new DagRuns. This inherently links Code Distribution (a file arriving on disk and being parsed) with Execution Activation (the scheduler and executor running that code).

For teams operating at enterprise scale, or utilizing large company-wide monorepos, this creates significant frictionleads to significant friction. For instance, this proposal comes from a team managing ~200k DAGs and version-pinning would be indispensable when a single python file generates hundreds of DAGs (or more) with a breaking change that affects only a subset of them. The DAG Processor runs on a continuous, non-deterministic loop. There is no guarantee exactly when new code synced to the server will be parsed and executed. This lack of determinism makes it difficult for DAG owners to plan changes where the cutover time is important. Additionally, the high latency of executing a standard git revert during a production outage also necessitates an instant rollback mechanism that can make Airflow go back to using an older DAG version.

This AIP proposes introducing a first-class execution primitive to explicitly control which version of a DAG is scheduled, decoupling file parsing from scheduling and execution.

Use cases

By decoupling the physical distribution of code from its execution, this primitive empowers users to achieve exact operational control over their DAGs:

• Strict Determinism for Cutovers: Pinning guarantees exact, to-the-second timing for business-critical logic cutovers (e.g., a financial reporting pipeline switching to next year's tax logic right after midnight on Jan 1st) driven by an external API call, rather than relying on an arbitrary parse loop cycle.

• Instant Incident Mitigation (MTTR): Rather than waiting for a Git revert to propagate through infrastructure (CI builds, monorepo merge, storage sync, parse loop), this proposal also enables programmatic, instant rollbacks. This capability also enables automations to be built on top of it. For example, if an observability tool detects DAG failures, it can instantly trigger an API call to revert the DAG to a previous version.

• Decouple Execution Window from Code Merge: Teams can safely merge and sync code as per their convenience without fear of immediate activation in production. An external release orchestrator can explicitly activate the new version programmatically when certain conditions are met, such as reaching a scheduled maintenance window.

Considerations

What change do you propose to make?

UI/UX Changes

DAG View

Add an "Active Version" field on the DAG view UI right next to the "Latest Dag Version" field.

A pencil icon next to it can open a dropdown with all available DAG versions along with a "Latest" option.

The default option for all DAGs will be "Latest" to preserve existing behavior. Internally, choosing the "Latest" option will clear the version pin (if any).

Database Schema Changes

Introduce a new nullable column to the dag table to track the active version.

• Table: dag

• New Column: active_dag_version_id (Integer, Nullable)

• Constraints: Foreign Key constraint referencing dag_version.id with ON DELETE SET NULL.

If this column is set, the scheduler uses the referenced version. If NULL, the system preserves existing Airflow behavior and defaults to the latest parsed version.

Core Scheduler and Execution Flow Changes

The scheduler's DAG run creation flow must be updated to respect the pinned version instead of defaulting to the latest bundle/version for new runs.

• Applicability of DAG version pin:

  • Changing the active_dag_version_id will not impact currently running DagRuns. They will safely continue executing on the version they started with (retaining existing AIP-63 behavior)

  • Task re-runs within an existing DagRun will continue to use the version associated with that DagRun

  • The pin only applies to the creation of new DagRuns and the expansion of new mapped tasks

• Update new DAG run creation flows to fetch the SerializedDagModel associated with the active version rather than the latest version

• Update task mapping fallbacks to get the active DAG version instead of the latest one

API changes

Introduce new endpoints under the dag_versions router to manage the execution lifecycle. These operations manipulate the execution state of a DAG and therefore require can_edit permissions on the specific DAG.

New: Pin a DAG Version for newer runs

POST /dags/{dag_id}/dagVersions/{dag_version_id}/pin

New: Unpin a DAG Version (Restore Default Behavior)

DELETE /dags/{dag_id}/dagVersions/pin

Existing: Expose Active Version in DAG Details

GET /dags/{dag_id}

Modification: Add active_dag_version_id to the DAGResponse schema.

Out of Scope

To ensure this feature remains a focused execution primitive, the following are explicitly out of scope:

• CI/CD Workflows: Airflow will not manage deployment stages, approval state machines, or testing environments. Pinning is simply an API-driven execution flag to be used by external orchestrators or end-users.

• Automated Reverts: Airflow will not automatically revert to previous versions upon task failures; this logic belongs in external observability/deployment tools.

Edge cases and other considerations

Listed below are some of the edge cases and considerations with this approach:

• Singleton Metadata Not Version-Pinned: The dag table fields always reflect the latest parsed code from the DAG Processor loop. If a DAG is pinned to v1, but v2 is parsed, some fields in DagModel will reflect v2. We may need to revisit the usages of some of these fields. It would be okay to use the latest version's values for many of the fields, but some usages might need to be updated to use the version-pinned SerializedDagModel for version-aware scheduling and execution behavior. For example, fields like owners, description, max_active_tasks, max_active_runs etc. may take effect immediately despite having a pinned DAG version. 

• Import Errors: A syntax error in the latest file sets has_import_errors = True on the DagModel. Currently, this halts scheduling for DAGs with import errors. This means a v2 file with import errors may block a healthy, pinned v1 DAG's execution until the error is fixed. The risk is low as import errors are preventable before code reaches production. Therefore, we accept this limitation for the scope of this AIP to avoid complex rewrites around the presence of import errors.

• DAG Deletion: If a new DAG bundle version removes a DAG, it will not be retained even if it's pinned to a DAG version. This can be considered as an informed and intended removal of the DAG.

• Asset Based Scheduling: The input and output asset associations and asset based schedule expressions need to be version-aware. Some of these considerations should've already been verified when the DAG versioning AIP was implemented.

• Database Performance: Introducing a version lookup on the scheduler hot-path could introduce minor overhead. This is mitigated by ensuring the usages efficiently use a DB index and leverage SQLAlchemy session's identity map and avoids issuing more queries during the scheduler loop.

• Triggers Not Version-Pinned: A trigger is just like a library dependency for DAGs and airflow services. Its implementation isn't pulled from a DAG bundle, instead it needs to be present on the triggerer service. So trigger definitions can't be versioned or pinned the same way as DAGs.

Which users are affected by the change?

• DAG owners: Gain the ability to quickly restore service during a bad DAG rollout via the UI or API without waiting for the Git revert lifecycle.

• Airflow administrators: Can now build external version control automations to programmatically control DAG version cutovers or rollbacks.

Users like data scientists, analysts, or developers who simply author DAGs will see no change. Their DAGs will continue to auto-execute the latest parsed version (the default NULL state) unless someone explicitly pins the DAG's version.

How are users affected by the change? (e.g. DB upgrade required?)

A minor database upgrade is required. API and UI users gain new execution lifecycle management tools.

What is the level of migration effort (manual and automated) needed for the users to adapt to the breaking changes? (especially in context of Airflow 3)

No manual effort. This change is purely additive. Database migrations are already automated.

Existing DAGs will default to NULL active version (execute latest), preserving the current Airflow behavior without intervention.

What defines this AIP as "done"?

• active_dag_version_id column added to the dag table.

• Scheduler code paths use the active DAG version for creating new runs without degrading core scheduling loop performance.

• Asset-aware scheduling is reviewed to work with the correct version pinning behavior.

• REST API endpoints (to pin and unpin DAG versions) are implemented, tested, and documented with correct RBAC enforcement.

• UI elements for managing and visualizing the active version are merged.

• Documentation is updated detailing the behavior of pinned versions vs. latest versions regarding singleton metadata (timetable, assets).