AIP-109 DAG Version Pinning

Status

Motivation

Airflow 3 introduces DAG Versioning (AIP-63: DAG Versioning), allowing the system to track the historical state of DAGs. However, the scheduler continues to automatically use the latest parsed DagVersion for all new DagRuns. This inherently links Code Distribution (a file arriving on disk and being parsed) with Execution Activation (the scheduler and executor running that code).

For teams operating at enterprise scale, or utilizing large company-wide monorepos, this leads to significant friction. For instance, this proposal comes from a team managing ~200k DAGs and version-pinning would be indispensable when a single python file generates hundreds of DAGs (or more) with a breaking change that affects only a subset of them. The DAG Processor runs on a continuous, non-deterministic loop. There is no guarantee exactly when new code synced to the server will be parsed and executed. This lack of determinism makes it difficult for DAG owners to plan changes where the cutover time is important. Additionally, the high latency of executing a standard git revert during a production outage also necessitates an instant rollback mechanism that can make Airflow go back to using an older DAG version.

This AIP proposes introducing a first-class execution primitive to explicitly control which version of a DAG is scheduled, decoupling file parsing from scheduling and execution.

Use cases

By decoupling the physical distribution of code from its execution, this primitive empowers users to achieve exact operational control over their DAGs:

• Strict Determinism for Cutovers: Pinning guarantees exact, to-the-second timing for business-critical logic cutovers (e.g., a financial reporting pipeline switching to next year's tax logic right after midnight on Jan 1st) driven by an external API call, rather than relying on an arbitrary parse loop cycle.

• Instant Incident Mitigation (MTTR): Rather than waiting for a Git revert to propagate through infrastructure (CI builds, monorepo merge, storage sync, parse loop), this proposal also enables programmatic, instant rollbacks. This capability also enables automations to be built on top of it. For example, if an observability tool detects DAG failures, it can instantly trigger an API call to revert the DAG to a previous version.

• Decouple Execution Window from Code Merge: Teams can safely merge and sync code as per their convenience without fear of immediate activation in production. An external release orchestrator can explicitly activate the new version programmatically when certain conditions are met, such as reaching a scheduled maintenance window.

Note that DAG version pinning can only be supported for DAGs loaded from versioned DAG bundles because this feature requires the ability to use an older bundle version during execution.

Considerations

What change do you propose to make?

UI/UX Changes

DAG View

Add an "Active Version" field on the DAG view UI right next to the "Latest Dag Version" field.

A pencil icon next to it can open a dropdown with all available DAG versions along with a "Latest" option.

The default option for all DAGs will be "Latest" to preserve existing behavior. Internally, choosing the "Latest" option will clear the version pin (if any).

Database Schema Changes

Introduce a new nullable column to the dag table to track the active version.

• Table: dag

• New Column: active_dag_version_id (uuid string, Nullable)

• Constraints: Foreign Key constraint referencing dag_version.id with ON DELETE SET NULL.

If this column is set, the scheduler uses the referenced version. If NULL, the system preserves existing Airflow behavior and defaults to the latest parsed version.

Alternatives - considered and rejected

• is_active boolean field in the dag_version table
  Difficult to represent the "only one active version per dag_id" constraint. Can't represent "use latest" simply.
• New dag_version_pin table
  Over-engineered for a single FK. Makes it necessary to perform joins for version lookups in the scheduler loop.

Core Scheduler and Execution Flow Changes

The scheduler's DAG run creation flow must be updated to respect the pinned version instead of defaulting to the latest bundle/version for new runs.

• Applicability of DAG version pin:

  • Changing the active_dag_version_id will not impact currently running DagRuns. They will safely continue executing on the version they started with (retaining existing AIP-63 behavior)

  • Task re-runs within an existing DagRun will continue to use the version associated with that DagRun

  • The pin only applies to the creation of new DagRuns

• Update new DAG run creation flows to fetch the SerializedDagModel associated with the active version rather than the latest version

• Update task mapping fallbacks to get the active DAG version instead of the latest one

API changes

Extend the PATCH /dags APIs to manage DAGs' active version. These operations manipulate the execution state of a DAG and therefore require can_edit permissions on the specific DAG.

Pin a DAG Version for newer runs

PATCH /dags/{dag_id}

{"active_dag_version_id": "9bc8a507-e1...049"}

Unpin DAG Version (Restore Default Behavior)

Single DAG

PATCH /dags/{dag_id}
{"active_dag_version_id": null}

Bulk API: Unpin version for all DAGs after applying the specified filters

PATCH /dags
{"active_dag_version_id": null}

Expose Active Version in DAG Details

GET /dags/{dag_id}

Modification: Add active_dag_version_id to the DAGResponse schema.

Note

• All of the APIs mentioned here already exist, they're just being updated.
• The bulk patch DAGs API cannot meaningfully pin the same version across multiple DAGs, so it will be restricted to only support unpinning an active version from DAGs
• To list all available versions for a given DAG, end-users and the UI can use the existing GET /dags/{dag_id}/dagVersions API

Out of Scope

To ensure this feature remains a focused execution primitive, the following are explicitly out of scope:

• CI/CD Workflows: Airflow will not manage deployment stages, approval state machines, or testing environments. Pinning is simply an API-driven execution flag to be used by external orchestrators or end-users.

• Automated Reverts: Airflow will not automatically revert to previous versions upon task failures; this logic belongs in external observability/deployment tools.

Edge cases and other considerations

Listed below are some of the edge cases and considerations with this approach:

• Singleton Metadata Not Version-Pinned: The dag table fields always reflect the latest parsed code from the DAG Processor loop. If a DAG is pinned to v1, but v2 is parsed, some fields in DagModel will reflect v2. We may need to revisit the usages of some of these fields. It would be okay to use the latest version's values for many of the fields, but some usages might need to be updated to use the version-pinned SerializedDagModel for version-aware scheduling and execution behavior. For example, fields like owners, description, max_active_tasks, max_active_runs etc. may take effect immediately despite having a pinned DAG version. 

• Import Errors: Syntax errors or performance issues while parsing a DAG file (slow or hung) sets has_import_errors = True on the DagModel. Currently, this halts scheduling for DAGs with import errors. This means a v2 file with import errors may block a healthy, pinned v1 DAG's execution until the error is fixed. The risk is low as import errors are preventable before code reaches production. Therefore, we accept this limitation for the scope of this AIP to avoid complex rewrites around the presence of import errors.

• DAG Deletion: If a new DAG bundle version removes a DAG, it will not be retained even if it's pinned to a DAG version. This can be considered as an informed and intended removal of the DAG.

• Asset Based Scheduling: The input and output asset associations and asset based schedule expressions need to be version-aware. Some of these considerations should've already been verified when the DAG versioning AIP was implemented.

• Database Performance: Introducing a version lookup on the scheduler hot-path could introduce minor overhead. This is mitigated by ensuring the usages efficiently use a DB index and leverage SQLAlchemy session's identity map and avoids issuing more queries during the scheduler loop.

• Triggers Not Version-Pinned: A trigger is just like a library dependency for DAGs and airflow services. Its implementation isn't pulled from a DAG bundle, instead it needs to be present on the triggerer service. So trigger definitions can't be versioned or pinned the same way as DAGs.

Which users are affected by the change?

• DAG owners: Gain the ability to quickly restore service during a bad DAG rollout via the UI or API without waiting for the Git revert lifecycle.

• Airflow administrators: Can now build external version control automations to programmatically control DAG version cutovers or rollbacks.

Users like data scientists, analysts, or developers who simply author DAGs will see no change. Their DAGs will continue to auto-execute the latest parsed version (the default NULL state) unless someone explicitly pins the DAG's version.

How are users affected by the change? (e.g. DB upgrade required?)

A minor database upgrade is required. API and UI users gain new execution lifecycle management tools.

What is the level of migration effort (manual and automated) needed for the users to adapt to the breaking changes? (especially in context of Airflow 3)

No manual effort. This change is purely additive. Database migrations are already automated.

Existing DAGs will default to NULL active version (execute latest), preserving the current Airflow behavior without intervention.

What defines this AIP as "done"?

• active_dag_version_id column added to the dag table.

• Scheduler code paths use the active DAG version for creating new runs without degrading core scheduling loop performance.

• Asset-aware scheduling is reviewed to work with the correct version pinning behavior.

• REST API changes to pin and unpin DAG versions are implemented, tested, and documented.

• UI elements for managing and visualizing the active version are merged.

• Documentation is updated detailing the behavior of pinned versions vs. latest versions regarding singleton metadata (timetable, assets).