The purpose of this document is to capture all the key elements which need to be done in order to release Airflow 2.0 to the world. This is a "living document" in that it is being updated every week as a result of ongoing progress.
Functional scope:
We are working to establish the scope of Airflow 2.0 so that it can be worked towards and released. The main functional elements which have been referenced and talked about by the Airflow PMC in various presentation as being components of Airflow 2.0 are listed below. The current scope as a result of the planning meetings in terms of big functional elements is:
- Scheduler HA - Improve Scheduler performance and reliability (AIP-15)
- Airflow REST API (AIP-32)
- Functional DAGs (AIP-31)
- Production-ready Docker Image and Helm Chart (AIP-26)
- Providers Package (AIP-21)
- Simplified Kubernetes Executor and KubePodOperator
- Smart Sensor PR (AIP-17) as an "early access" feature.
The following functional elements were discussed and deferred to a later (post 2.0) release:
- DAG Versioning
- Schedule Interval / Execution at Start of Schedule or End of Schedule
The following functional elements were discussed as a possibility for 2.0:
- Improvements to SubDAGs (AIP-34): Task Groups as a replacement for SubDags
Next steps
The immediate action items with respect to scope and major milestones are:
- Merge the Scheduler HA code into master and 2.0.0-test branch. This is in progress as we speak.
- Finish the discussion regarding versioning on the Providers Packages and take action as needed
- Discuss next steps in the mailing list about the "TaskGroup as a replacement for SubDags (AIP-34)". There was discussion around deprecating SubDags as part of 2.0, but maybe early to do so.
- Finalize the status and next steps around the Helm Chart part of AIP-26 above.
- Finish up the REST API discussions around permissions for AIP-32. It seems very close with a PR in review, but this needs to close in the next week.
- Scope out the documentation additions and revisions needed for Airflow 2.0 release.
The major upcoming milestones are:
Date | Milestone |
---|---|
Week of 7 Sep 2020 | Create the 2.0.0-test branch – DONE |
While the scope is fluid, we would be rebasing this test branch from master. After we completely freeze the scope, we would only cherrypick commits from Airflow Master to v2-0-test branch if they are “in-scope”. Normal development would continue on Master branch i.e. PRs would be created against Airflow Master. | |
Week of 28 Sep 2020 | Cut Functionally complete 2.0 alpha release |
Week of 12 Oct 2020 | Cut first 2.0 beta release |
Beta snapshots would be published to the Airflow Community to test and create issues to make sure Airflow is functioning and backwards compatible. | |
Week of 19 Oct 2020 | Cut bridge release based on 1.10.x - jump-off point to 2.0. Probably 1.10.13 or 1.10.14 containing upgrade check scripts for 2.0 |
Week of 26 Oct 2020 | Cut second 2.0 beta release |
Week of 9 Nov 2020 | Cut third 2.0 beta release |
Week of 23 Nov 2020 | Cut first 2.0 release candidate (2.0.0rc1) |
As part of creating the release candidate, an official vote would be started to release 2.0.0. If there are any bugs discovered in 2.0.0rc1, they will be fixed and a new release candidate will be published. This will be repeated until the Vote passes |
Approach
In the interest of clarity, the core approach is detailed below:
- Semantic versioning
Since Airflow has adopted Semantic versioning, this is the opportunity to make significant changes to Airflow, including deprecating functionality and breaking changes. - Deprecate, don’t break
However, the key principle should be to deprecate functionality but not to break existing functionality. - Backwards compatibility
As far as possible, this release should be backwards compatible. If it is not possible to be backwards compatible, this should be flagged and ideally automatically migrated through a utility (which could be an add-on).
Non-functional scope:
We also need to establish the non-functional elements needed for a “major release” such as this. These include:
- Capturing changes which are needed:
- Connections (making them unique)
- Changes to be made to the configuration file
- Changes to be made to DAGs / SubDags?
- Changes to the installation mechanism: Providers, Plugins etc.
- Changes to any other core concepts
- Migration tools:
- “Am I ready” to migrate to Airflow 2.0?
- Upgrade configurations, etc. from 1.xx to 2.0
- Manual changes to migrate/upgrade
- Migrating from “experimental” to new API
- Testing
- Docs
- Updating.md guide especially needs to cover what the migration script does and more comprehensive and easily readable
- Make docs-site cleaner and more organized
Github Issues:
To track the progress of work, we use:
- Meta-issue: Release Airflow 2.0 #10152
- Milestone: "Airflow 2.0" milestone
Dev Calls
To agree, stay aligned to the process and track the progress of Airflow 2.0 we should have regular calls between Airflow Develops (PMC Members, Committers and whoever is willing to help in Airflow 2.0).
Calendar: