Page History

This page describes a proposed The purpose of an Airflow Improvement Proposal (AIP) process for proposing a major change to Airflow.is to introduce any major change to Apache Airflow. This is required in order to balance the need to support new features, use cases, while avoiding accidentally introducing half thought-out interfaces that cause needless problems when changed.

What is considered a major change that needs an AIP?

Any of the following should be considered a major change:

• Any major new feature, subsystem, or piece of functionality
• Any change that impacts the public interfaces of the project

What are the "public interfaces" of the project? All of the following are public interfaces that people build around:

• Airflow DAG DSL components: DAG, XCom, TaskInstance and context.
• Web UI
• Core operators (BaseOperator, SubDagOperator, SensorOperator)
• [Airflow 2.0] REST endpoints
• Command line tools and arguments

Proposal lifecycle

Discuss

Most proposals start with an idea. If you have an idea of how Airflow could improve, we encourage you to send an email to dev@airflow.apache.org with a subject starting with [DISCUSS]. This email thread will allow you to gather early feedback. We encourage you to start a Draft document or share a Google Document with the mailing list detailing your proposal. Use AIP template here to file details on your proposal.

Draft

Anyone is welcome to propose a new Improvement to Airflow. You will need to request edit access to the Confluence page in order to create your AIP document. You can do so by sending an email with your Wiki ID to dev@airflow.apache.org and request permission.

To create your own AIP, click on Create AIP. If you don't have permission, please send an email with your Wiki ID to dev@airflow.incubator.apache.org and request permission.

Adopted AIPs

Vote

Once you or someone else feels like there’s a rough consensus on the idea and there’s no strong opposition, you can move your proposal to the Vote phase. For this you will send a new email to the dev list with the subject starting with [VOTE]. This phase follows a vote on code modifications for Apache projects as described here.  Remember to link the discussion thread to your AIP and ideally also to this vote email. And don’t forget to link the AIP to the email as well.

The voting has to last at least 72 hours and it’s a good practice to extend this time if most of this period is a weekend. Whole community is encouraged to give +1 and -1 votes. There’s also fractional votes (0.5, -0.99) that allow you to express your view on the discussion without giving an explicit approval/block signal. However, only Airflow committers and PMC members have a binding vote. This means that any vote requires at least 3, +1 binding votes to be recognized as passed. In addition, any -1 binding vote will move the proposal back to the Discuss phase in order to address changes. 

The vote is a consensus-seeking process, “the goal is not to *win* votes or come to a unanimous agreement, but rather to ensure that there’s a forum for people to raise and discuss their concerns, and that nobody feels strongly enough to block the group from moving forward. Consensus-seeking emphasizes discussion over enumeration: *Rough consensus is achieved when all issues are addressed, but not necessarily accommodated.*” (quote from Working in Public: The Making and Maintenance of Open Source Software by Nadia Eghbal)

If you want to introduce significant changes to your proposal after it has been accepted you can do so following the same procedure.

Accepted

Congratulations! Your proposal has been accepted. Next up is breaking down tasks. You should break down tasks on Apache Airflow GitHub issues page. You can optionally group the tasks together in a Github Project if it consists of multiple steps and you want to be able to track progress for its implementation or share the progress with others. Once issues are created, make sure to send them to the vote thread (optionally Airflow slack). Ask a committer to create a label for your AIP and link all your issues with it for easier searchability. 

Now you can start contributing and - more importantly - you can, and probably should, encourage others to contribute to your project. However this is optional and up to your discretion. You can learn how to contribute and communicate  by reading the Contributing guide in the GitHub repository. Use your imagination and various communication channels for ways to encourage people if you do not already have contributors following your idea. It is advisable that several people work on the AIP so that the knowledge is shared.

Completed

You should move your AIP to complete after you consider the main bulk of work has been merged to the repository. It is okay to leave open issues for minor follow up tasks like adding additional capabilities or similar. 

Abandoned

AIP may be moved to abandoned if there has been no owner or intention of developing it further for over a year or if the owner decides to abandon it. The latter can happen if the creator decides to refactor AIP into a new AIP for example or if there is a proposal that supersedes the original AIP.

AIPs under discussion

AIPs under discussion

Accepted AIPs

AIPs which have been voted on and accepted for inclusion in Airflow, but not yet (fully) implemented.

Completed AIPs

"Umbrella" AIPs

AIPs hat are too big to be treated as "deliverable" AIPs on its own - they are split into smaller AIPs.

...

Abandoned AIPs

...

...

...

...

...

...

(The above list is automatically generated based on page labels. Do not edit manually)

...