Page History

The purpose of this document is to capture all the key elements which need to be done in order to release Airflow 2.0 to the world. The scope and timing shown here is a "statement of intent", primarily for transparency and visibility to the community about the goals for Airflow 2.0 and the progress being made. This is a "living document" in that it is being updated every week as a result of ongoing progress. 

Functional scope: 

We need to establish have effectively finalized the scope of Airflow 2.0 so that it can be worked towards and and now actively workings towards merging all the code and getting it released. The main big functional elements which have been referenced and talked about by the Airflow PMC in various presentation as being components of Airflow 2.0 are:are listed below:

1. Scheduler HA - Improve Scheduler performance and reliability (AIP-15)Scheduler HA 
2. Airflow REST API
3. DAG Versioning
4. (AIP-32)
5. Functional DAGs (AIP-31)
6. Production-ready Docker Image & Helm Chart(AIP-26)
7. Providers Package (AIP-21)
8. Simplified Kubernetes Executor and KubePodOperator
9. Smart Sensor PR (AIP-17) as an "early access" feature. 
10. Improvements to SubDAGs (AIP-34): Task Groups as an alternative to SubDAGs and a future replacement

The following functional elements were discussed and deferred to a later (post 2.0) release:

• DAG Versioning
• Schedule Interval / Execution at Start of Schedule or End of Schedule
• Production-ready Helm Chart (part of AIP-26)

The current status of the above in-scope items with respect to code needed to be merged into master:

The major upcoming milestones are detailed below. As stated above, please note that these are statements of intent, rather than absolute.

Approach

In the interest of clarity, the core approach is detailed below:

1. Semantic versioning
   Since Airflow has adopted Semantic versioning, this is the opportunity to make significant changes to Airflow, including deprecating functionality and breaking changes. 
   
   
2. Deprecate, don’t break 
   However, the key principle should be to deprecate functionality but not to break existing functionality. 
   
   
3. Backwards compatibility 
   As far as possible, this release should be backwards compatible. If it is not possible to be backwards compatible, this should be flagged and ideally automatically migrated through a utility (which could be an add-on). 

Non-functional scope: 

We also need to establish the non-functional elements needed for a “major release” such as this. These include.

One of the most important elements here is to make the upgrade from Airflow 1.10.x to Airflow 2.0 as straight forward as possible. This includes the following elements:

1. Capturing all the changes which are needed :Schedule (Setting to add choice of schedule at end or schedule at start of interval)by the Airflow user in making the transition:
   
   • Connections (making in them unique)
   • etc
   • Changes to be made to the configuration file
   • Changes to be made to DAGs
/ SubDags ?
2. • Changes to the installation mechanism: Providers, Plugins etc.
3. Changes to any other core concepts
4. Migration toolsUpgrade check scripts:
• “Am I ready” to migrate to Airflow 2.0?
• Upgrade configurations, etc. These scripts are targeted from 1.xx .10.13 the "bridge release" to 2.0
5. Manual changes to migrate / upgrade
• Migrating from “experimental” to new API
6. Testing
   
7. Docs
• Updating.md guide especially needs to cover what the migration script does and more comprehensive and easily readable
• Make docs-site cleaner and more organized

High-level milestones forward:

At a high level, here the key steps which need to be done: 

1. Agree on scope
2. Identify the depth of completeness of the high-level items (Current status)
3. Identify what all needs to be get done for both functional and non-functional scope
4. Functionally complete
5. Non-functional scope complete
6. Validation complete
7. Beta release of Airflow 2.0
8. Production release of Airflow 2.0

Process:

...

1. Migration tools / scripts:
   • Ideally some scripts which can enable automation in the process of upgrading from Airflow 1.10.13 to 2.0, to reduce the possibility of errors. 
2. Upgrade to 2.0 docs:
   • Create a separate "Upgrade to 2.0" document and split it from Updating.md guide. 
   • This will cover both what is covered in the upgrade checks scripts as well as other manual changes required either immediately or over time
3. Unavoidable manual changes to take advantage of new functionality
• Migrating from the “experimental” API to the new REST API. The experimental API will be deprecated and migration to the new REST API is strongly encouraged.
• However, the new REST API implements authorization (similar to the UI), so invocation of the API will be a little different from before.

Since this is a significant body of work, the "upgrade check scripts" and the "migration tools" will be versioned independently of the core Airflow release, so that these utilities can be improved on an ongoing basis, without requiring an Airflow release. We anticipate that these utilities will continue to improve even after Airflow 2.0 is released, based on feedback. 

The other important element which is being worked on parallel is improving the overall documentation around Airflow including changes to make the docs-site cleaner and more organized. That effort will be ongoing and is not covered here as part of the Airflow 2.0 plan. 

Github Issues:

To track the progress of work, we use:

• Meta-issue: Release Airflow 2.0 #10152
• Milestone: "Airflow 2.0" milestone 

Release definitions

As requested, Alpha and Beta release thinking is elaborated here:

Alpha release: 

• Integrated build with all main features included.
• Intended for the developer community:
  • To be used to make sure that there are no regressions and that all the new features added are still working as expected
  • To be used to perform end to end testing
  • To be used to perform benchmarks to be published as feasible
• Not intended for end users, since this may still contain serious errors and instability. 

Beta release:

• Functionally complete and all features frozen.
• Has been tested by the developer community and no critical issues are open. 
• Beta release is stable and can be used by end user community in "real world" scenarios, though production usage is not recommended.
• Ideally, should include guides to cover changes in setup, system migration, and DAG migration from 1.10.x. 
• Benchmarks to be run and published.

...

Dev Calls

To agree, stay aligned to the process and track the progress of Airflow 2.0 we should have regular calls between Airflow Develops (PMC Members, Committers and whoever is willing to help in Airflow 2.0).

Calendar Link: 

...