[DRAFT] AIP-100 Airflow Scheduler Performance Improvement

Status

Motivation

The Airflow scheduler, a core airflow component, is the most important part of airflow, as it handles all the interactions with tasks, dagruns and the executor, meaning that it has a lot of responsibilities.

Some of the major responsibilities include (listed in the order they are listed in the SchedulerJobRunner code):

1. Create airflow "runnable units"
   1. Create all dagruns needed to be created for each dag, which is a critical section per dag row.
      1. This includes time based scheduling and asset based scheduling
   2. Start and schedule all queued dagruns (move them from the queue state to running).
   3. Create new tasks (incl mapped tasks) if the dag has changed.
2. Schedule and run tasks
   1. deciding which tasks to run, move the tasks to queued state.
   2. move tasks to executors and upate their state.
3. Manage the running of the task (syncing the executor)
   
   1. Monitor all running tasks.
   2. update the states of running tasks.

Most of the logic lies in the second major responsibility, being, scheduling and running tasks.

This responsibility is the one which takes the most amount of time, and where the improvements will make the most impact.

Improving task throughput and reducing starvation of tasks, will benefit all workflows, and reducing the lag and total dagrun time, by reducing the scheduled tasks time.

Considerations

What change do you propose to make?

What problem does it solve?

Why is it needed?

Are there any downsides to this change?

Which users are affected by the change?

How are users affected by the change? (e.g. DB upgrade required?)

What is the level of migration effort (manual and automated) needed for the users to adapt to the breaking changes? (especially in context of Airflow 3)

N/A

Other considerations?

What defines this AIP as "done"?

The Airflow scheduler's performance improves, by increasing task throughput and allowing multiple schedulers to run workloads simultaneously, increasing the benefit of scaling out the scheduler.