Status

Motivation

The Airflow scheduler is a core component, which makes it very sensitive to change, meaning that even improvements need to undergo very thorough checks and reviews, that makes it hard to push new improvements and fixes to the scheduler.

There cannot be 1 scheduler that will work for all possible cases and workloads, due to the flexibility that airflow allows.

Allowing the scheduler class to change, even as an external package, may open a lot of new possibilities, without hindering the flexibility that airflow currently gives, allowing for alternative schedulers and experiments in the scheduler to run, without interfering with or being directly under the airflow package, lowering the maintenance burden from the main airflow repository, and using it as community packages, just like is possible with executors in airflow.

Considerations

There are a few considerations about the given AIP, mainly, from the security perspective, as the scheduler is a core component which has full read and write access to the db, and a bad actor may be able to exploit to gain access to secrets.

Will the 'external scheduler' be considered a trusted or not, as it did not come from the airflow repository, as this will change the implementation design of the feature. 

What change do you propose to make?

Implementing a flag in the 'airflow scheduler' command to allow the change of the class that the command runs in order to start up the scheduler, allowing to change all or part of the implementations of the scheduler by extending the existing SchedulerJobRunner class, the command to run such a scheduler should look as such: 'airflow scheduler --impl "com.my.scheduler.NewScheduler

Chart changes are also needed, to allow for the deployment of a custom scheduler implementation selection (a string for python package / class), with control over how many custom schedulers to run (to allow for safe blue/green like deployment).

A deployment with a custom implementation will have the following values:

scheduler:
  custom:
    enabled: true
    package: "com.my.scheduler.NewScheduler"
    instances: 2

What problem does it solve?

As of now, developing on the scheduler has to happen very delicately, and thoroughly checked, with countless manual tests that need to be run by the person who wants to make a change, a good example is pr #54103, where a change that should increase the schedulers efficiency is proposed, it required the author (alone) to run tests and collect metrics, if such a feature to allow for plugability of the scheduler existed, the scheduler could be developed as a third party module, tested by the community, which would allow for increased efficiency in testing and development phase, and would provide more insights than a single person can on a change to core airflow scheduler logic.

Why is it needed?

Developing new features for the scheduler tends to require heavy testing work (in addition to approval from project members), which is usually done by one person, the one who is developing the feature, which leaves a lot of room for error, as the developer has to deploy and entire cluster to check the scheduler under production workflows.

An example could be making sleep operator not take up an executor slot, and becoming a "special case" just like empty operator, this requires changes to the scheduler core logic, which may hinder certain workflows, but will improve a workflow that uses the sleep operator a lot, in this case, having a plugable scheduler, and having control over how many of the schedulers run as the custom implementation would make testing the new logic much simpler, as the developer could enroll it into his production environment and gradually increase the count of schedulers which have the new feature, allowing for proper testing, catching more edge cases and having a proper full scale proof of concept for the new feature, aiding in the decision of approving such changes, as they may already have community approval.

Due to the wide variety of workflows that are run using airflow, the current scheduler might not work well for all of the use cases, sometimes improvements are proposed, yet they need to be checked with all existing workflows and the specific workflows they are optimizing for.

Are there any downsides to this change?

• Since the scheduler is a core component, it requires to remain stable, introducing this feature, may cause more work to be done on the scheduler, resulting in, possibly, more mishaps of the scheduler breaking, even after all the tests (more changes = higher chance for breakage).
• Development of third party schedulers may cause delays in airflow's release cycle (due to airflow being a community lead project) as if the community widely uses a certain scheduler and a change is introduced which causes it to stop working, the change might be delayed by the community until the other scheduler is fixed.
• When conflicting features arise, it might be hard to decide which one will be part of the airflow main repo, especially if both are equally supported by the community.

Which users are affected by the change?

The change affects only the following roles:

• Deployment Managers:  when deploying the instance, they should choose whether to add a custom scheduler, and how many of those should be deployed.
• Airflow Developers: who may want to add a feature to the scheduler, can now test it easily without too much risk.

How are users affected by the change? (e.g. DB upgrade required?)

Users should not be affected by the change, the change will not affect the end users by default, only if the implementation is changed, only then will the users feel a change (i.e scheduling decisions, new concurrency limits, etc) .

What is the level of migration effort (manual and automated) needed for the users to adapt to the breaking changes? (especially in context of Airflow 3)

N/A

Other considerations?

• Should, as part of the feature, we allow the user to run more than 1 custom scheduler implementation?
• Should this feature be labeled as production use applicable or development only?

What defines this AIP as "done"?

It is possible to change the implementation of part or all of the scheduler instances, allowing for alternate custom tailored implementations and allowing for faster exploration and development of features.