Motivation
The critical section in Airflow's scheduler uses an optimistic strategy: it fetches a batch of scheduled task instances from the database (up to max_tis_per_query), then filters them in Python against all concurrency limits:
- pool slots
max_active_tasks_per_dagmax_active_tis_per_dag,max_active_tis_per_dagrun- executor slots (exception as it's local to scheduler).
This works well enough for standard workloads. However, in large-scale deployments with thousands of tasks—often driven by dynamic task mapping—it causes starvation, where the scheduler fixates on tasks from one constrained group, discards most after checks, and queues far fewer than possible per cycle. The issue appears identically across every one of these limits. It was first noted with prioritized pools nearly full, starving lower-priority ones despite free slots (Issue 45636). Unknown User (xbis) highlighted the same with huge DAGs hitting max_active_tasks. (Mail discussion). The pattern repeats for symmetrically for other concurrency limits: instead of skipping to viable tasks, the scheduler loops over the same ineligible set.
Narrow fixes like PR 54103 tackle one limit but ignore the rest, and production environments differ widely—from DAG run floods to mapped task surges or complex priorities. Building all limits into the initial query has been tough, often infeasible in the current design. PR 53492 tried window functions and lateral joins for a full solution, but orthogonal limits (pools independent of DAG caps) led to poor SQL performance and unresolved edges. PR 55537 utilized pessimistic DB procedures, which looks promising as it solves the problem, but still needs thorough examination and community discussion. Rethinking the priorities in airflow seems to be the simplest method, not yet examined practically (no POC created), where dag-level priorities are introduced, while the current task-level priorities will become internal to each dagrun, this methodology relies on changing the task-creation method, creating only tasks we can run with all concurrency limits considered per dagrun, and simplifying the main scheduler loop, fetching only max_tis tasks once, and moving them to the executor, in a single iteration, as any task in the scheduled state, is guaranteed to be able to run without further checks (other than executor_slots ).
Large-scale reliability is the goal here. Dynamic mapping enables massive DAGs for per-item batching (one task per file or data pointer), so Airflow has to handle it without breaking. Community benchmarks on real heavy loads would help confirm solutions, and we need queuing that's truly resistant to starvation under any concurrency limit. This proposal aims to serve the updated summary of our research in improving the scheduler's throughput and eliminating starvation. As there's still uncertainty regarding the desired outcome, we present all the strategies we tried or thought of, and expect a fruitful discussion.
Considerations
What change do you propose to make?
Change the methodology in which the scheduler currently decides which tasks to run, allowing for an increased throughput of the scheduler and running more tasks with a wider variety of workflows without causing noisy neighbours during scheduling.
Why is it needed?
Large-scale Airflow deployments with dynamic task mapping—generating thousands of tasks per DAG—suffer severe scheduler starvation. The optimistic strategy fetches batches of tasks but discards most due to concurrency limits, queuing almost nothing per cycle while workers idle and tasks pile up in scheduled state indefinitely. This change ensures scalable, equitable queuing across all limits, avoiding partial fixes and enabling reliable performance at extreme volumes.
Are there any downsides to this change?
Promising solutions (like SQL procedures) apparently impose a maintenance overhead which has to be considered while deciding on a change. Some other solutions require significant refactoring of the scheduler's logic which requires resources and dedication.
Which users are affected by the change?
- Large-scale deployments
- Deployments shared by several teams
- Power users - clients that heavily utilize concurrency limits and priority weights
How are users affected by the change? (e.g. DB upgrade required?)
A DB upgrade might be required, depending on the solution of choice.
- Single-limit pre-filtering:
- No db migration is needed.
- Window functions:
- 2 integer fields need to be added to the task_instance field, nullable and slowly inserted as tasks are created.
- SQL Procedures:
- Same as window functions.
- Priority Semantic Changes:
- Possibly adding a priority field to the dag and or dagrun table.
Alternatives considered
| Strategy | Current optimistic | Single-limit pre-filtering | Window functions | SQL procedures / exhaustive linear scan | Priority Semantic Changes |
| Description | Fetch up to | Use a single limit, like max_active_tasks and eliminate starvation on that limit using lateral join / window function. | Nested window functions/lateral joins incorporate multiple limits in a single query. | DB-stored procedures (PostgreSQL/MySQL) or Python (SQlite) scan sorted TIs exhaustively until enough queueable ones found. | Increase the scheduler's variety of choice by giving up the concept of task-level priorities in favor of more global, DAG-level priorities. |
| Motivation | Simple, fast for small/medium workloads; no complex SQL. | Improves performance by considering just one most common concurrency limit which works for many users. | Theoretically, could allow checking multiple concurrency limits in a single SQL query. | Universal solution for all orthogonal limits; eliminates post-fetch drops entirely. Runs on DB side and prevents RTT latencies / slow Python processing. | Priorities make it very hard to come up with a viable in-code solution as every single task should be considered in each iteration. Allowing for a change in the priorities defenition (which as of now is not clearly defined) will allow us to consider tasks in a simple manner without causing a maintnance overhead while solving starvation |
| Advantages | Minimal query overhead; works across DBs including SQLite; easy to maintain/debug. | Boosts throughput for large DAGs (hundreds of tasks); avoids wasting cycles re-querying saturated DAGs. | Theoretically comprehensive, single query for all DB vendors. | Throughput gain even with multiple limits, fewest scheduler iterations, blazingly-fast linear scanning. Solves all starvation cases by using a simple and algorithmically efficient scan on DB side where the data resides. | Per-task priorities are incomprehensible and significantly complicate scheduling. Directly removing any scheduling strategy relying on obsolescence, meaning round-robin style scheduling is not possible. Using an obsolescence based priority algorithm allows for a clearly defined priority behaviour, while keeping scheduling descisions effiecient and fair. |
| Drawbacks | Performance degradation that leads to starvation. | Considers just one limit and solves just part of the problem, while some limits continue to starve. For example, while max_active_tasks is handled in the query, pool limits may cause tasks to be dropped. | Window functions' behavior is not suitable for Airflow task concurrency model, and they don't solve all cases while imposing a large performance overhead. | Requires several implementations for different DB vendors, may be considered a technical debt. The logic is difficult to test, and SQlite doesn't support procedures at all. | Requires a complete refactor of priority logic which is a breaking change. Redefining priorities in a way such that we have dag-level priorities and inter-dagrun priorities will cause a breakage in the current expected behaviour, along with obsolescence, it may vary how clusters behave. Defining a good obsolescence scheduling heuristic can be hard and tedious, as it requires trial and error. |
| Drawback mitigations | Deploy multiple Airflow clusters for large workloads. | It solves the common case of DAG run level limit. Don't define complex concurrency constraints in workloads. | Cases where tasks are dropped with this approach are rare. | DB vendors are not added too often, as critical section algorithm isn't frequently updated, if at all. If it is, adding the logic in several places isn't a big overhead. Keep the logic as simple as possible so no testing is needed. Mimic the algorithm in Python for SQlite. | The change is beneficial to the project as it both contributes to solving starvation and provides a more useful priority feature, while allowing to control which workflows are more important than the others, without starving other workflows. |
| Summary | Serves typical workloads well but fundamentally incapable of handling large-scale starvation without repeated post-fetch drops. | Helpful short-term throughput boost for one common limit, but can't be a long-term solution since other concurrency limits like pools and max_active_tis_per_dag/dagrun remain frequently used and broken. | Theoretically elegant but ruled out—unsuitable for Airflow's multi-dimensional concurrency model and imposes a slight performance overhead (up to 10x slower on large tables). | A viable path forward; benchmarks show 1.7-2x throughput gains across single/dual/triple limit workloads. Remaining technical issues (metrics, SQLite, DB fields) are resolvable, drawback mitigation is possible. | Promising alternative if community reaches consensus on desired priority semantics; enables simpler in-code round-robin scheduling by reducing per-task priority complexity. |
| POC | Runs in production | PR 54103 | PR 53492 | PR 55537 | None |
Implementation Considerations
Priority Semantic Refactor
After a discussion in the community, there seems to form a strong opinion that Rethinking the priorities is the most viable option out of the bunch.
All the given solutions are the simplest to implement when implemented at task creation, as you can query once per scheduler loop and create only tasks which will be able to run, accounting for concurrency limits, while simplifying the current scheduler loop, where there will be only 1 iteration of task querying from the db, and moving those tasks straight to the respective executor, without any additional checks (other than executor checks).
Allowing for a round-robin style scheduling descisions to be made with more room for freedom, and reducing the places in which starvation can occurr.
Rethinking priorities leaves the door open for ideas and proposals, so far the current proposals and Ideas we have thought about are as follows.
| Proposal | Introducing dag-level priorities, while keeping current priorities | Keeping current priorities while changing their meaning | Completely removing all priorities at all levels |
|---|---|---|---|
| Explanation | Introducing dag-level priorities, where each dagrun created, inherits the priority of the dag from which it was created. Dag-level priority acts as a global priority deciding which dags will be looked at first. Keeping current task priorities, yet changing their semantics, where now task-priorities are inter-dagrun. | Running over the dagruns with a round-robin loop based on Task priorities are global where the dagruns are also sorted by task priority after Or Task priorities decide only inter-dagrun scheduling decisions. | Removing all priorities, and using plain round-robin scheduling by `last_scheduling_decision` of the dagrun. Checking only the concurrency limits, and creating a task without looking or caring about ordering. |
| Motivation | Users still may want to prioritize entire workflows over others, while also allowing for finer grained prioritization. All of that with no user facing API breakage. | No database changes required, keeping as much of the current behaviour, while solving starvation. | A single database change is required, yet is the simplest way to implement given solution, as it requires the least amount of code changes. |
| Advantages | Does not break the user facing api, while allowing for fine grained priority tuning. Is simple to implement and maintain, where relatively minor changes are done to the task-creation logic, while simplifying the scheduler class. | No api breakage. | Simplest and easiest to implement and maintain, no starvation due to priorities as they simply do not exist. |
| Drawbacks | TBD. | Harder to implement, as it requires more complex sql queries to be remain with the current behaviour. Due to tasks or operators being saved in the database as a json field in the | A major API breakage, which is not backwards compatible, and removes a feature which has existed for a long time in airflow. |
| Summary | An approach with the least destructive changes, while solving the issue. | No api breakage, with a higher maintnance burden and more implementation complexity. | Simplest implementation and maintnance, yet removes a widely used feature. |
Obsolescence Algorithm
Implementing any change in the priority semantics still leaves place for starvation, where same prioritized dagruns are considered over and over, without considering other lower-priority dagrun's tasks, even if we can run the given tasks, and so in this case, an obsolescence algorithm comes into play.
Having an obsolescence algorithm means that even if there are more prioritized dagruns, their relative priority changes, as we take into a count the time since last scheduling descision on the current dagruns, meaning that the older the dagrun, the higher priority it will be, while a higher priority dagrun which is old, will still be looked at before a lesser prioritized dagrun waiting for the same amount of time.
This introduces the complication of choosing the obsolescence algorithm, such that it remains simple, easy to implement and understand while still allowing for round-robin style prioritizd scheduling.
What defines this AIP as "done"?
The Airflow scheduler's performance improves in terms of queued tasks over a time frame. Any unnecessary starvation is eliminated after improving the algorithmics of the task queueing logic.
Appendix
Considerations will be moved here after a decision is made on the strategy.