Page History

Status

Motivation

The Airflow scheduler is , a core component, which makes it very sensitive to change, meaning that even improvements need to undergo very thorough checks and reviews, that makes it hard to push new improvements and fixes to the scheduler.

There cannot be 1 scheduler that will work for all possible cases and workloads, due to the flexibility that airflow allows.

Allowing the scheduler class to change, even as an external package, may open a lot of new possibilities, without hindering the flexibility that airflow currently gives, allowing for alternative schedulers and experiments in the scheduler to run, without interfering with or being directly under the airflow package, lowering the maintenance burden from the main airflow repository, and using it as community packages, just like is possible with executors in airflow.

airflow component, is the most important part of airflow, as it handles all the interactions with tasks, dagruns and the executor, meaning that it has a lot of responsibilities.

Some of the major responsibilities include (listed in the order they are listed in the SchedulerJobRunner code):

1. Create airflow "runnable units"
   1. Create all dagruns needed to be created for each dag, which is a critical section per dag row.
      1. This includes time based scheduling and asset based scheduling
   2. Start and schedule all queued dagruns (move them from the queue state to running).
   3. Create new tasks (incl mapped tasks) if the dag has changed.
2. Schedule and run tasks
   1. deciding which tasks to run, move the tasks to queued state.
   2. move tasks to executors and upate their state.
3. Manage the running of the task (syncing the executor)
   
   1. Monitor all running tasks.
   2. update the states of running tasks.

Most of the logic lies in the second major responsibility, being, scheduling and running tasks.

This responsibility is the one which takes the most amount of time, and where the improvements will make the most impact.

Improving task throughput and reducing starvation of tasks, will benefit all workflows, and reducing the lag and total dagrun time, by reducing the scheduled tasks time.

Considerations

What change do you propose to make?

What problem does it solve?

Why is it needed?

Are there any downsides to this change?

Which users are affected by the change?

Considerations

There are a few considerations about the given AIP, mainly, from the security perspective, as the scheduler is a core component which has full read and write access to the db, and a bad actor may be able to exploit to gain access to secrets.

Will the 'external scheduler' be considered a trusted or not, as it did not come from the airflow repository, as this will change the implementation design of the feature. 

What change do you propose to make?

Implementing a flag in the 'airflow scheduler' command to allow the change of the class that the command runs in order to start up the scheduler, allowing to change all or part of the implementations of the scheduler by extending the existing SchedulerJobRunner class, the command to run such a scheduler should look as such: 'airflow scheduler --impl "com.my.scheduler.NewScheduler

Chart changes are also needed, to allow for the deployment of a custom scheduler implementation selection (a string for python package / class), with control over how many custom schedulers to run (to allow for safe blue/green like deployment).

A deployment with a custom implementation will have the following values:

scheduler:
  custom:
    enabled: true
    package: "com.my.scheduler.NewScheduler"
    instances: 2

What problem does it solve?

As of now, developing on the scheduler has to happen very delicately, and thoroughly checked, with countless manual tests that need to be run by the person who wants to make a change, a good example is pr #54103, where a change that should increase the schedulers efficiency is proposed, it required the author (alone) to run tests and collect metrics, if such a feature to allow for plugability of the scheduler existed, the scheduler could be developed as a third party module, tested by the community, which would allow for increased efficiency in testing and development phase, and would provide more insights than a single person can on a change to core airflow scheduler logic.

Why is it needed?

Developing new features for the scheduler tends to require heavy testing work (in addition to approval from project members), which is usually done by one person, the one who is developing the feature, which leaves a lot of room for error, as the developer has to deploy and entire cluster to check the scheduler under production workflows.

An example could be making sleep operator not take up an executor slot, and becoming a "special case" just like empty operator, this requires changes to the scheduler core logic, which may hinder certain workflows, but will improve a workflow that uses the sleep operator a lot, in this case, having a plugable scheduler, and having control over how many of the schedulers run as the custom implementation would make testing the new logic much simpler, as the developer could enroll it into his production environment and gradually increase the count of schedulers which have the new feature, allowing for proper testing, catching more edge cases and having a proper full scale proof of concept for the new feature, aiding in the decision of approving such changes, as they may already have community approval.

Due to the wide variety of workflows that are run using airflow, the current scheduler might not work well for all of the use cases, sometimes improvements are proposed, yet they need to be checked with all existing workflows and the specific workflows they are optimizing for.

Are there any downsides to this change?

• Since the scheduler is a core component, it requires to remain stable, introducing this feature, may cause more work to be done on the scheduler, resulting in, possibly, more mishaps of the scheduler breaking, even after all the tests (more changes = higher chance for breakage).
• Development of third party schedulers may cause delays in airflow's release cycle (due to airflow being a community lead project) as if the community widely uses a certain scheduler and a change is introduced which causes it to stop working, the change might be delayed by the community until the other scheduler is fixed.
• When conflicting features arise, it might be hard to decide which one will be part of the airflow main repo, especially if both are equally supported by the community.

Which users are affected by the change?

The change affects only the following roles:

• Deployment Managers:  when deploying the instance, they should choose whether to add a custom scheduler, and how many of those should be deployed.
• Airflow Developers: who may want to add a feature to the scheduler, can now test it easily without too much risk.

How are users affected by the change? (e.g. DB upgrade required?

)

What is the level of migration effort (manual and automated) needed for the users to adapt to the breaking changes? (especially in context of Airflow 3)

N/A

Other considerations

?

What defines this AIP as "done"?

It is possible to change the implementation of part or all of the scheduler instances, allowing for alternate custom tailored implementations and allowing for faster exploration and development of featuresThe Airflow scheduler's performance improves, by increasing task throughput and allowing multiple schedulers to run workloads simultaneously, increasing the benefit of scaling out the scheduler.