DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
Motivation
The critical section in Airflow
scheduler, a core airflow component, is the most important part of airflow, as it handles all the interactions with tasks, dagruns and the executor, meaning that it has a lot of responsibilities.Some of the major responsibilities include (listed in the order they are listed in the SchedulerJobRunner code):
- Create airflow "runnable units"
- Create all dagruns needed to be created for each dag, which is a critical section per dag row.
- This includes time based scheduling and asset based scheduling
- Start and schedule all queued dagruns (move them from the queue state to running).
- Create new tasks (incl mapped tasks) if the dag has changed.
- Create all dagruns needed to be created for each dag, which is a critical section per dag row.
- Schedule and run tasks
- deciding which tasks to run, move the tasks to queued state.
- move tasks to executors and upate their state.
- Manage the running of the task (syncing the executor)
- Monitor all running tasks.
- update the states of running tasks.
Most of the logic lies in the second major responsibility, being, scheduling and running tasks.
This responsibility is the one which takes the most amount of time, and where the improvements will make the most impact.
As clusters scale, there are more tasks, and more limits, with more varying workloads, meaning that at some point, hoping to get enough tasks ready for execution, which causes all workloads to run slower, due to being in the scheduled state for longer, rather than running as soon as all execution limits allow us to do so.
's scheduler uses an optimistic strategy: it fetches a batch of scheduled task instances from the database (up to max_tis_per_query), then filters them in Python against all concurrency limits:
- pool slots
max_active_tasks_per_dagmax_active_tis_per_dag,max_active_tis_per_dagrun- executor slots (exception as it's local to scheduler).
This works well enough for standard workloads. However, in large-scale deployments with thousands of tasks—often driven by dynamic task mapping—it causes starvation, where the scheduler fixates on tasks from one constrained group, discards most after checks, and queues far fewer than possible per cycle. The issue appears identically across every one of these limits. It was first noted with prioritized pools nearly full, starving lower-priority ones despite free slots (Issue 45636). Unknown User (xbis) highlighted the same with huge DAGs hitting max_active_tasks. The pattern repeats for symmetrically for other concurrency limits: instead of skipping to viable tasks, the scheduler loops over the same ineligible set.
Narrow fixes like PR 54103 tackle one limit but ignore the rest, and production environments differ widely—from DAG run floods to mapped task surges or complex priorities. Building all limits into the initial query has been tough, often infeasible in the current design. PR 53492 tried window functions and lateral joins for a full solution, but orthogonal limits (pools independent of DAG caps) led to poor SQL performance and unresolved edges. PR 55537 utilized pessimistic DB procedures, which looks promising as it solves the problem, but still needs thorough examination and community discussion.
Large-scale reliability is the goal here. Dynamic mapping enables massive DAGs for per-item batching (one task per file or data pointer), so Airflow has to handle it without breaking. Community benchmarks on real heavy loads would help confirm solutions, and we need queuing that's truly resistant to starvation under any concurrency limit.
This proposal aims to serve the updated summary of our research in improving the scheduler's throughput and eliminating starvation. As there's still uncertainty regarding the desired outcome, we present all the strategies we tried or thought of, and expect a fruitful discussionImproving task throughput and reducing starvation of tasks, will benefit all workflows, and reducing the lag and total dagrun time, by reducing the scheduled tasks time.
Considerations
What change do you propose to make?
What problem does it solve?
Why is it needed?
Are there any downsides to this change?
Which users are affected by the change?
How are users affected by the change? (e.g. DB upgrade required?)
What is the level of migration effort (manual and automated) needed for the users to adapt to the breaking changes? (especially in context of Airflow 3)
N/A
Other considerations?
What defines this AIP as "done"?
The Airflow scheduler's performance improves, by increasing task throughput and allowing multiple schedulers to run workloads simultaneously, increasing the benefit of scaling out the scheduler.