You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 17 Next »

The purpose of this document is to capture all the key elements which need to be done in order to release Airflow 2.0 to the world. The scope and timing shown here is a "statement of intent", primarily for transparency and visibility to the community about the goals for Airflow 2.0 and the progress being made. This is a "living document" in that it is being updated every week as a result of ongoing progress. 

Functional scope: 

We have effectively finalized the scope of Airflow 2.0 and now actively workings towards merging all the code and getting it released. The big functional elements are listed below:

  1. Scheduler HA - Improve Scheduler performance and reliability (AIP-15)
  2. Airflow REST API (AIP-32)
  3. Functional DAGs (AIP-31)
  4. Production-ready Docker Image (AIP-26)
  5. Providers Package (AIP-21)
  6. Simplified Kubernetes Executor and KubePodOperator
  7. Smart Sensor PR (AIP-17) as an "early access" feature. 
  8. Improvements to SubDAGs (AIP-34): Task Groups as an alternative to SubDAGs and a future replacement

The following functional elements were discussed and deferred to a later (post 2.0) release:

  • DAG Versioning
  • Schedule Interval / Execution at Start of Schedule or End of Schedule
  • Production-ready Helm Chart (part of AIP-26)


The current status of the above in-scope items with respect to code needed to be merged into master:

Scheduler HA (AIP-15)PRs have been created against master and in review.
REST API (AIP-32)Mostly merged. Small additions remain.
Functional DAGs (AIP-31)Merged.
Providers Package (AIP-21)Most of the work done already as part of Backport Providers. Open items need to be addressed.
Simplified Kubernetes Executor and KubePodOperatorMerged
Smart Sensors (AIP-17)Merged
Task Groups - Improvements to SubDAGs (AIP-34)Merged

Next steps

The immediate action items with respect to scope and major milestones are:

  1. Scheduler HA: Reviews need to be finished and then the PRs merged.
  2. REST API Permissions: Reviews need to be completed and the PRs merged.
  3. The discussion around versioning on the Providers Packages is in under progress. PRs need to be created and then merged. 
  4. Scope out and create the urgent documentation additions needed for the beta testing of the Airflow 2.0 release. 


The major upcoming milestones are detailed below. As stated above, please note that these are statements of intent, rather than absolute.

DateMilestone
Week of 7 Sep 2020Create the 2.0.0-test branch – DONE

While the scope is fluid, we would be rebasing this test branch from master. After we completely freeze the scope, we would only cherrypick commits from Airflow Master to v2-0-test branch if they are “in-scope”. Normal development would continue on Master branch i.e. PRs would be created against Airflow Master.

Week of 28 Sep 2020Cut Functionally complete 2.0 alpha release
Week of 12 Oct 2020Cut first 2.0 beta release

Beta snapshots would be published to the Airflow Community to test and create issues to make sure Airflow is functioning and backwards compatible. 

Week of 19 Oct 2020Cut bridge release based on 1.10.x - jump-off point to 2.0. Probably 1.10.13 or 1.10.14 containing upgrade check scripts for 2.0
Week of 26 Oct 2020Cut second 2.0 beta release
Week of 9 Nov 2020Cut third 2.0 beta release
Week of 23 Nov 2020Cut first 2.0 release candidate (2.0.0rc1)

As part of creating the release candidate, an official vote would be started to release 2.0.0. If there are any bugs discovered in 2.0.0rc1, they will be fixed and a new release candidate will be published.  This will be repeated until the Vote passes

Approach

In the interest of clarity, the core approach is detailed below:

  1. Semantic versioning
    Since Airflow has adopted Semantic versioning, this is the opportunity to make significant changes to Airflow, including deprecating functionality and breaking changes. 

  2. Deprecate, don’t break 
    However, the key principle should be to deprecate functionality but not to break existing functionality. 

  3. Backwards compatibility 
    As far as possible, this release should be backwards compatible. If it is not possible to be backwards compatible, this should be flagged and ideally automatically migrated through a utility (which could be an add-on). 


Non-functional scope: 

We also need to establish the non-functional elements needed for a “major release” such as this. These include:

  1. Capturing changes which are needed:
    • Connections (making them unique)
    • Changes to be made to the configuration file
    • Changes to be made to DAGs / SubDags?
    • Changes to the installation mechanism: Providers, Plugins etc.
    • Changes to any other core concepts
  2. Migration tools:
    • “Am I ready” to migrate to Airflow 2.0?
    • Upgrade configurations, etc. from 1.xx to 2.0
  3. Manual changes to migrate/upgrade
    • Migrating from “experimental” to new API
  4. Testing
  5. Docs
    • Updating.md guide especially needs to cover what the migration script does and more comprehensive and easily readable
    • Make docs-site cleaner and more organized

Github Issues:

To track the progress of work, we use:

Dev Calls

To agree, stay aligned to the process and track the progress of Airflow 2.0 we should have regular calls between Airflow Develops (PMC Members, Committers and whoever is willing to help in Airflow 2.0).

Calendar: 


  • No labels