Status

Motivation

The Airflow scheduler is a core component, which makes it very sensitive to change, meaning that even improvements need to undergo very thorough checks and reviews, that makes it hard to push new improvements and fixes to the scheduler.

There cannot be 1 scheduler that will work for all possible cases and workloads, due to the flexibility that airflow allows.

Allowing the scheduler class to change, even as an external package, may open a lot of new possibilities, without hindering the flexibility that airflow currently gives, allowing for alternative schedulers and experiments in the scheduler to run, without interfering with or being directly under the airflow package, lowering the maintenance burden from the main airflow repository, and using it as community packages, just like is possible with executors in airflow.

Considerations

There are a few considerations about the given AIP, mainly, from the security perspective, as the scheduler is a core component which has full read and write access to the db, and a bad actor may be able to exploit to gain access to secrets.

Will the 'external scheduler' be considered a trusted or not, as it did not come from the airflow repository, as this will change the implementation design of the feature. 

What change do you propose to make?

What problem does it solve?

Why is it needed?

Are there any downsides to this change?

Which users are affected by the change?

How are users affected by the change? (e.g. DB upgrade required?)

Users should not be affected by the change, the change will not affect the end users by default, only if the implementation is changed, only then will the users feel a change (i.e scheduling decisions, new concurrency limits, etc) .

What is the level of migration effort (manual and automated) needed for the users to adapt to the breaking changes? (especially in context of Airflow 3)

Example: DAG changes needed by the DAG Author. Can this be automated via Migration utilities, if so, how?

Other considerations?

What defines this AIP as "done"?

It is possible to change the implementation of part or all of the scheduler instances, allowing for alternate custom tailored implementations and allowing for faster exploration and development of features.