DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
Status
Motivation
The critical section in Airflow's scheduler uses an optimistic strategy: it fetches a batch of scheduled task instances from the database (up to max_tis_per_query), then filters them in Python against all concurrency limits:
- pool slots
max_active_tasks_per_dagmax_active_tis_per_dag,max_active_tis_per_dagrun- executor slots (exception as it's local to scheduler).
This works well enough for standard workloads. However, in large-scale deployments with thousands of tasks—often driven by dynamic task mapping—it causes starvation, where the scheduler fixates on tasks from one constrained group, discards most after checks, and queues far fewer than possible per cycle. The issue appears identically across every one of these limits. It was first noted with prioritized pools nearly full, starving lower-priority ones despite free slots (Issue 45636). Unknown User (xbis) highlighted the same with huge DAGs hitting max_active_tasks. The pattern repeats for symmetrically for other concurrency limits: instead of skipping to viable tasks, the scheduler loops over the same ineligible set.
Narrow fixes like PR 54103 tackle one limit but ignore the rest, and production environments differ widely—from DAG run floods to mapped task surges or complex priorities. Building all limits into the initial query has been tough, often infeasible in the current design. PR 53492 tried window functions and lateral joins for a full solution, but orthogonal limits (pools independent of DAG caps) led to poor SQL performance and unresolved edges. PR 55537 utilized pessimistic DB procedures, which looks promising as it solves the problem, but still needs thorough examination and community discussion.
Large-scale reliability is the goal here. Dynamic mapping enables massive DAGs for per-item batching (one task per file or data pointer), so Airflow has to handle it without breaking. Community benchmarks on real heavy loads would help confirm solutions, and we need queuing that's truly resistant to starvation under any concurrency limit.
This proposal aims to serve the updated summary of our research in improving the scheduler's throughput and eliminating starvation. As there's still uncertainty regarding the desired outcome, we present all the strategies we tried or thought of, and expect a fruitful discussion.
Considerations
What change do you propose to make?
Switch the scheduler's critical section from optimistic post-fetch filtering to a pessimistic strategy where the initial database query respects all concurrency limits upfront. This ensures only queueable tasks are retrieved, fully resolving starvation across large-scale workloads via a configurable, pluggable implementation.
Why is it needed?
Large-scale Airflow deployments with dynamic task mapping—generating thousands of tasks per DAG—suffer severe scheduler starvation. The optimistic strategy fetches batches of tasks but discards most due to concurrency limits, queuing almost nothing per cycle while workers idle and tasks pile up in scheduled state indefinitely. This change ensures scalable, equitable queuing across all limits, avoiding partial fixes and enabling reliable performance at extreme volumes.
Are there any downsides to this change?
Promising solutions (like SQL procedures) apparently impose a maintenance overhead which has to be considered while deciding on a change. Some other solutions require significant refactoring of the scheduler's logic which requires resources and dedication.
Which users are affected by the change?
Large-scale, multi-tenant deployments, power users - clients that heavily utilize concurrency limits and priority weights.
How are users affected by the change? (e.g. DB upgrade required?)
A DB update is required in most known solutions to add.
Alternatives considered
| Strategy | Current optimistic | Single-limit pre-filtering | Window functions | SQL procedures / exhaustive linear scan | Refactor priorities |
| Description | Fetch up to | Use a single limit, like max_active_tasks and eliminate starvation on that limit using lateral join / window function. | Nested window functions/lateral joins incorporate multiple limits in a single query. | DB-stored procedures (PostgreSQL/MySQL) or Python (SQlite) scan sorted TIs exhaustively until enough queueable ones found. | Give up the concept of task-level priorities in favor of more global, DAG-level priorities. |
| Motivation | Simple, fast for small/medium workloads; no complex SQL. | Improves performance by considering just one most common concurrency limit which works for many users. | Theoretically, could allow checking multiple concurrency limits in a single SQL query. | Universal solution for all orthogonal limits; eliminates post-fetch drops entirely. Runs on DB side and prevents RTT latencies / slow Python processing. | Priorities make it very hard to come up with a viable in-code solution as every single task should be considered in each iteration. |
| Advantages | Minimal query overhead; works across DBs including SQLite; easy to maintain/debug. | Boosts throughput for large DAGs (hundreds of tasks); avoids wasting cycles re-querying saturated DAGs. | Theoretically comprehensive, single query for all DB vendors. | Throughput gain even with multiple limits, fewest scheduler iterations, blazingly-fast linear scanning. Solves all starvation cases by using a simple and algorithmically efficient scan on DB side where the data resides. | Per-task priorities are incomprehensible and significantly complicate scheduling, eliminating every round-robin based strategy. By reducing their importance we can come up with a more efficient scheduling algorithm. |
| Drawbacks | Performance degradation that leads to starvation. | Considers just one limit and solves just part of the problem. | Window functions' behavior is not suitable for Airflow task concurrency model and don't solve all cases while imposing a large performance overhead. | Requires several implementations for different DB vendors, may be considered a technical debt. The logic is difficult to test, and SQlite doesn't support procedures at all. | Requires a complete refactor of priority logic which is a breaking change. |
Other considerations?
What defines this AIP as "done"?
The Airflow scheduler's performance improves in terms of queued tasks over a time frame. Any unnecessary starvation is eliminated after improving the algorithmics of the task queueing logic.