THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
The purpose of this document is to capture all the key elements which need to be done in order to release Airflow 2.0 to the world. The scope and timing shown here is a "statement of intent", primarily for transparency and visibility to the community about the goals for Airflow 2.0 and the progress being made. This is a "living document" in that it is being updated every week as a result of ongoing progress.
Functional scope:
We have effectively finalized the scope of Airflow 2.0 and now actively workings towards merging all the code and getting it released. The big functional elements are listed below:
- Scheduler HA - Improve Scheduler performance and reliability (AIP-15)
- Airflow REST API (AIP-32)
- Functional DAGs (AIP-31)
- Production-ready Docker Image (AIP-26)
- Providers Package (AIP-21)
- Simplified Kubernetes Executor and KubePodOperator
- Smart Sensor PR (AIP-17) as an "early access" feature.
- Improvements to SubDAGs (AIP-34): Task Groups as an alternative to SubDAGs and a future replacement
...
| Scheduler HA (AIP-15) | Merged. |
| REST API (AIP-32) | Merged. |
| Functional DAGs (AIP-31) | Merged. |
| Providers Package (AIP-21) | Most of the work done already as part of Backport Providers. Some release management items such as: Versioning, package creation, documentation handling are openMerged. Being released independently as intended. |
| Simplified Kubernetes Executor and KubePodOperator | Merged |
| Smart Sensors (AIP-17) | Merged |
| Task Groups - Improvements to SubDAGs (AIP-34) | Merged |
...
The immediate action items with respect to scope and major milestones are:
- Cut the first beta build.
- Encourage users to test the build build, especially around Kubernetes.
- Work on the 1.10.13 "bridge release" to cut it and enable migration / upgrade testing.
- Accelerate the Documentation for "Getting started with 2.0 and upgrading to 2.0" to enable beta testing, as well as for 2.0 once released. Also, enhance the upgrade check scripts as needed.
- Close out release management related open items with respect to release of Providers packages. Not necessarily needed for beta1, but before RC.
The major upcoming milestones are detailed below. As stated above, please note that these are statements of intent, rather than absolute.
| Date | Milestone | ||
|---|---|---|---|
| Week of 7 Sep 2020 | Create the 2.0.0-test branch - Done. | ||
While the scope is fluid, we would be rebasing this test branch from master. After we completely freeze the scope, we would only cherrypick commits from Airflow Master to v2-0-test branch if they are “in-scope”. Normal development would continue on Master branch i.e. PRs would be created against Airflow Master. | |||
| Week of 12 Oct 2020 | First Airflow 2.0 alpha release "apache-airflow-2.0.0.a1" cut on Oct 13th. | ||
| Week of 26 Oct 2020 | Second Airflow 2.0 alpha release "apache-airflow-2.0.0.a2" cut on Oct 26th. | ||
| Week of 9 Nov 2020 | Cut first 2.0 beta release. . First beta build cut on Nov 9th. | ||
Beta snapshots would be published to the Airflow Community to test and create issues to make sure Airflow is functioning and backwards compatible outside of known changes. | |||
| 10 Nov - 7 Dec | Cut beta releases approximately every Tuesday. Latest beta build is beta3, cut on Nov 18th and available for testing. | ||
| Weeks of 16 Nov and 23 Week of 16 Nov 2020 | Cut bridge release based on 1.10.x - jump-off point to 2.0 . Probablyand submit for vote. Release candidate 1.10. 13 or 1.10.14 containing upgrade check scripts for 2.013rc1 submitted for vote on Nov 20th. Release the 1.10.13 "bridge release" after vote is concluded. Released on 25th Nov. Upgrade check scripts released independently. Release candidate of Upgrade check scripts cut and submitted for vote on Nov 20th. Release the upgrade check scripts after vote is concluded. Airflow Upgrade Checks 1.0.1 released on 25th Nov. | ||
| Week of 30 Nov 2020 | Cut bridge release 1.10.14 to resolve bug found in 1.10.13 after release and submit for vote. Expected release the following week | 10 Nov - 7 Dec | Cut beta releases approximately every Tuesday . |
| Week of 7 Dec 2020 | Cut first 2.0 release candidate (2.0.0rc1) and submit for vote. Airflow Providers release 1.0.0rc1 cut and submitted for vote on 9th Dec. Airflow 1.10.14 released on 10th Dec. Airflow Upgrade Check release 1.1.0rc1 cut and submitted for vote on 11th Dec. | ||
As part of creating the release candidate, an official vote of 3 working days would be started to release 2.0.0. If there are any bugs discovered in 2.0.0rc1, they will be fixed and a new release candidate will be published. This will be repeated until the Vote passes. | |||
| Week of 14 Dec 2020 | Based on votes passing, officially release Airflow 2.0.0, the Providers packages, and the Upgrade scripts. Due to bugs found, we had to cut additional release candidate builds. Airflow 2.0.0rc3 is now the latest release candidate. This was submitted for vote on December 14th. Airflow 2.0.0 released on December 17! |
Approach
In the interest of clarity, the core approach is detailed below:
- Semantic versioning
Since Airflow has adopted Semantic versioning, this is the opportunity to make significant changes to Airflow, including deprecating functionality and breaking changes. - Deprecate, don’t break
However, the key principle should be to deprecate functionality but not to break existing functionality. - Backwards compatibility
As far as possible, this release should be backwards compatible. If it is not possible to be backwards compatible, this should be flagged and ideally automatically migrated through a utility (which could be an add-on).
Non-functional scope:
We also need to establish the non-functional elements needed for a “major release” such as this.
...
The other important element which is being worked on parallel is improving the overall documentation around Airflow including changes to make the docs-site cleaner and more organized. That effort will be ongoing and is not covered here as part of the Airflow 2.0 plan.
Github Issues:
To track the progress of work, we use:
- Meta-issue: Release Airflow 2.0 #10152
- Milestone: "Airflow 2.0" milestone
Release definitions
As requested, Alpha and Beta release thinking is elaborated here:
...
- Functionally complete and all features frozen.
- Has been tested by the developer community and no critical issues are open.
- Beta release is stable and can be used by end user community in "real world" scenarios, though production usage is not recommended.
- Ideally, should include guides to cover changes in setup, system migration, and DAG migration from 1.10.x.
- Benchmarks to be run and published.
Dev Calls
To agree, stay aligned to the process and track the progress of Airflow 2.0 we should have regular calls between Airflow Develops (PMC Members, Committers and whoever is willing to help in Airflow 2.0).
...