Status

Motivation

Real-world pipelines rarely fit a clean "all-or-nothing" dependency model. A few recurring shapes that Airflow users encounter every day:

• Fan-out data ingestion. A team ingests partitioned data from 40 source systems in parallel, then runs a single aggregation task downstream. It is acceptable, and often expected, for a few partitions to fail on any given run (a source is temporarily down, a third-party API rate-limits). The aggregation should still proceed as long as, say, at least 35 partitions succeeded.
• ML training sweeps. An ML engineer trains 20 candidate models in parallel and a downstream task picks the best one. The downstream task should run when all 20 finish and at least 3 produced a valid model; anything less means the sweep itself failed and the downstream is pointless.
• Data quality gates. A platform team runs dozens of quality checks in parallel before allowing a publish step. The publish should proceed only when all checks finished and no more than one was skipped (and zero failed), with "skipped" treated as inconclusive.
• Multi-region replication. A job copies a dataset to N regions. A downstream notification task should fire when all copies are done, and at least a configured quorum succeeded.

None of these are exotic. They show up in analytics platforms, ML infrastructure, observability pipelines, and data platform teams across industries. They share the same shape: the decision to run a downstream task depends on counts and ratios of upstream outcomes, not on every upstream reaching one specific state.

Airflow's current TriggerRule is a flat enum of 13 hand-picked presets. When one of those presets happens to match a real-world shape, authoring is easy. When it does not, Dag authors fall back to one of three workarounds, each with real business cost:

1. A branching operator upstream of the real work. Adds a node to the Dag whose only job is to paper over a trigger-rule gap. The graph becomes harder to read, and a Dag-level concern (when should this task run?) becomes a task-level concern (compute a branch decision).
2. Handle it inside the downstream task itself. The task starts, inspects upstream TaskInstance states via the metadata DB, and either continues or short-circuits. This breaks the normal model where state transitions live in the scheduler, makes retries and SLAs behave oddly, and couples Dag logic to Airflow internals.
3. Accept a weaker approximation and suppress the problem. Use ALL_SUCCESS and let a tolerable failure fail the whole branch, or use ALL_DONE and let a bad upstream silently poison the downstream. Teams catch this in production, not in code review.

The core problem is that the set of useful shapes is combinatorial. Covering it with a preset enum means the enum grows every time a new shape is needed. The pattern does not scale: each new preset needs an enum value, an evaluator branch, tests, documentation, and a release cycle before authors can use it. Meanwhile the authoring story for Dag authors stays the same: wait for a preset or work around the gap.

This AIP proposes an additive authoring surface that lets Dag authors describe the condition they actually want, in one line, without waiting for a new preset. The 13 existing enum values remain valid and unchanged; they simply become shortcuts for specific expressions.

Considerations

• No change to failure semantics. Every existing enum value maps to exactly one expression. For every upstream-state vector, the enum form and its expression equivalent must produce the same downstream outcome: same pass or fail, same resulting TaskInstance state. This is the AIP's primary correctness gate.
• Backward compatible. All 13 existing enum values keep working. String literals like trigger_rule="all_success" keep working. No DB migration, no Dag rewrites required.
• Scheduler and Dag version skew. A Dag authored with the new expression form and then read by a scheduler that predates this AIP must fail cleanly at Dag-load time, not silently fall back to a different rule. The wire format carries an explicit version field so future evolutions stay safe.
• Adoption complexity. Airflow's current trigger rules are simple and predictable; this proposal makes them more powerful, which is inherently less predictable. The mitigation is documentation: enums stay the primary example in tutorials, and the expression form is presented as the tool for cases where no enum fits.

What change do you propose to make?

 Add a structured expression form for trigger rules that lives alongside the existing enum.

from airflow.sdk import TriggerRule as TR

# Today's form, still valid
task_a = EmptyOperator(task_id="a", trigger_rule=TR.ALL_DONE_MIN_ONE_SUCCESS)

# New equivalent
task_b = EmptyOperator(
    task_id="b",
    trigger_rule=TR.expr(done="all", success=">=1"),
)

# Combinations that previously required a new enum value become one-liners
task_c = EmptyOperator(
    task_id="c",
    trigger_rule=TR.expr(done="all", success=">=35"),   # ingestion fan-out, 35 of 40 partitions
)

task_d = EmptyOperator(
    task_id="d",
    trigger_rule=TR.expr(failed=0, upstream_failed=0, skipped="<=1"),
)

The expression accepts keyword thresholds against the upstream-state counts Airflow already computes (success ,failed, skipped , upstream_failed, removed, done, plus the setup-scoped variants used for setup and teardown). Values accept "all" , "none" , "any"  or ">=1" , plain integers, or comparison strings (">=N", "<=N", "==N", "<N", ">N"). Conditions are implicitly ANDed.

Out of scope

The following are intentionally excluded from this AIP. They are viable extensions that could be proposed as separate AIPs or follow-up PRs once the expression form is in users' hands and real feedback is available:

• Per-upstream named conditions. Expressing "task A must succeed, task B just needs to be done" on individual upstream edges (for example, a >> Require("success") >> downstream). This introduces composition semantics with the task-level rule that deserve their own discussion.
• Boolean operators. OR-combined conditions (for example, "at least one success OR all skipped") and conditional or implication logic (for example, "if A succeeded, then B must too"). The aggregate form here is AND-only by design.
• Fractional thresholds. Specifying targets as ratios rather than counts (for example, "at least 50% succeeded"). Useful but a larger design surface.
• Arbitrary Python predicates. Letting authors plug in a callable to decide whether to proceed. Power-user territory with significant operational implications.
• Evaluator consolidation refactor. Collapsing the existing if-elif chain in the trigger-rule evaluator into a single dispatch over preset expressions is an internal simplification this AIP unlocks, but it is not a user-visible change and can be proposed as a follow-up PR.

What problem does it solve?

Authors today express dependency conditions by picking from a hard-coded list of 13 presets. When the one they need is not there, the options are:

1. Propose and land a new enum value, which can take months between idea and general availability.
2. Shoehorn the logic with a branching operator, adding a node to the Dag whose only purpose is to cover a trigger-rule limitation.
3. Accept a weaker approximation and handle the remainder in-task, breaking Airflow's state-transition model.

All three carry real cost: slower iteration, more brittle Dags, harder post-mortems when a pipeline misbehaves because "the branch operator was doing something clever". The expression form replaces the preset-hunting step with a direct statement of what the author actually wants, in the Dag file, with no release cycle involved.

There is a secondary internal benefit. The evaluator today is a large if-elif chain, one branch per enum value. Once the expression form is in place, each enum value's branch can collapse to "look up the preset expression and evaluate it". That is a smaller surface area to maintain and a safer place to add future extensions.

Why is it needed?

The problem is not a specific missing preset. It is the trajectory. Every team that needs a combination not currently in the enum repeats the same loop: file an issue, write a PR, wait for the release, then adopt the rule. The set of useful combinations is large enough (and grows with the ecosystem) that chasing it through enum additions is structurally the wrong approach.

Concretely, requirements along the lines of the following have come up over the past two years, and each would have needed its own enum value under the current model:

• "All done, at least two succeeded."
• "None failed, at most one skipped."
• "All done, exactly one failed."
• "At least N of M succeeded" for varying N and M.

Each is one line of expression and zero enum churn under this proposal.

Are there any downsides to this change?

• More surface area to document - The expression form has its own small grammar. The docs need a clear mapping table and a "when to use an enum vs an expression" note.
• Authors may overuse it - A rule like `TR.expr(success="all")` is just `ALL_SUCCESS` written the long way. Docs should steer toward using the enum when it already fits.
• Readability drift in code review - A reviewer looking at `trigger_rule=TR.expr(failed=0, upstream_failed=0, skipped=0, done="all")` has to mentally evaluate what it means, versus `NONE_FAILED`. Good-faith use of the enum where it fits mitigates this.
• Serialization-format evolution - Adding a structured field to the serialized Dag means mixed-version clusters (new scheduler with old webserver, or vice versa) need a clear failure mode. Addressed via a mandatory version field and a hard parse error on unknown formats, but the contract has to be honored going forward.

Which users are affected by the change?

• Dag authors - gain a new authoring surface. Existing Dags are unaffected; nothing changes unless a Dag opts in by using `TR.expr(...)`.
• Operational users - structured aggregate trigger rules should be reflected in the UI

How are users affected by the change? (e.g. DB upgrade required?)

- No DB migration - Trigger rules live in serialized Dag JSON, which is regenerated every parse cycle. No schema changes, no upgrade scripts.
- No Dag changes required - Nothing breaks. Authors choose when, or whether, to adopt the expression form.

What is the level of migration effort (manual and automated) needed for the users to adapt to the breaking changes? (especially in context of Airflow 3)

No migration is required - This is a strictly additive change; no existing behavior is altered, deprecated, or removed. Dag authors do not need to rewrite anything.

If an author chooses to adopt the expression form for an existing Dag, the change is a one-line substitution per task. The mapping table above functions as a drop-in translation. An automated rewrite (via a `ruff`-style codemod or a small script) would be straightforward, but is not required: the enum form remains fully supported and is often more readable when it fits.
There are no breaking changes in this AIP. The existing enum values are not being deprecated or removed. Whether to deprecate any of them (for example, `ALL_DONE_MIN_ONE_SUCCESS` once its expression form is canonical) is explicitly a decision for a future AIP, not this one.

Other considerations?

• Interaction with setup and teardown - Setup and teardown tasks have hard-coded rule behaviors (ALL_DONE_SETUP_SUCCESS on teardowns, implicit ALL_SUCCESS on work tasks downstream of setups). This AIP does not touch those semantics. The expression form is a drop-in replacement for ordinary trigger rules and does not unlock or alter setup or teardown behavior.

What defines this AIP as "done"?

• The expression form is available in the Task SDK, documented with the mapping table, and has at least one example Dag in example_dags/.
• Every existing TriggerRule enum value has a corresponding expression, verified by a parametrised test matrix: for every upstream-state vector in the existing test suite, the enum form and the expression form produce the same downstream outcome.
• Serialized Dags round-trip cleanly: a Dag authored with TR.expr(...) serializes, deserializes, and evaluates to the same behavior on subsequent scheduler cycles.
• The wire format carries an explicit version field, and deserializers raise a clear Dag-load error when encountering an unknown format, verified by test.
• The UI renders the structured form in the task details panel.
• A short "when to use which form" note is added to the trigger rules documentation.