Versions Compared

Old Version 13

changes.mady.by.user Vikram Koka

Saved on

compared with

New Version 14

changes.mady.by.user Vikram Koka

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Updated based on planning meeting yesterday. Consolidated high level milestones and process sections into the timeline table

...

We are working to establish the scope of Airflow 2.0 so that it can be worked towards and released. The main functional elements which have been referenced and talked about by the Airflow PMC in various presentation as being components of Airflow 2.0 are listed below. The current scope as a result of the planning meetings in terms of big functional elements areis:

  1. Scheduler HA - Improve Scheduler performance and reliability (AIP-15)
  2. Airflow REST API (AIP-32)
  3. Functional DAGs (AIP-31)
  4. Production-ready Docker Image and Helm Chart (AIP-26)
  5. Providers Package (AIP-21)
  6. Simplified Kubernetes Executor and KubePodOperator
  7. Smart Sensor PR (AIP-17) as an "early access" feature. 

...

The immediate action items now with respect to scope are:

  1. Finish the post-discussion action items on the Providers Packages. 
  2. Discuss progress and details on API and Improvements to SubDags (AIP-34)

...

The major upcoming milestones are:

DateMilestone
Week of

...

7

...

Sep 2020Create the 2.0.0-test branch

While the scope is fluid, we would be rebasing this test branch from master. After we completely freeze the scope, we would only cherrypick commits from Airflow Master to v2-0-test

...

branch if they are “in-scope”. Normal development would continue on Master branch i.e. PRs would be created against Airflow Master.

Week of

...

28

...

Sep 2020

...

Cut Functionally complete 2.0 alpha release
Week of 12 Oct 2020Cut first 2.0 beta release

Beta snapshots would be published to the Airflow Community to test and create issues to make sure Airflow is functioning and backwards compatible. 

Week of 19 Oct 2020Cut bridge release based on 1.10.x - jump-off point to 2.0. Probably 1.10.13 or 1.10.14 containing upgrade check scripts for 2.0
Week of

...

26 Oct 2020Cut second 2.0 beta release
Week of 9 Nov 2020Cut third 2.0 beta release
Week of 23 Nov 2020Cut first 2.0 release candidate (2.0.0rc1)

As part of creating the release candidate, an official vote would be started to release 2.0.0. If there are any bugs discovered in 2.0.0rc1, they will be fixed and a new release candidate will be published.  This will be repeated until the Vote passes

Approach

In the interest of clarity, the core approach is detailed below:

...

  1. Capturing changes which are needed:
    • Schedule (Setting to add choice of schedule at end or schedule at start of interval)
    • Connections (making in unique)etc
    • Changes to be made to the configuration file
    • Changes to be made to DAGs / SubDags ?
    • Changes to the installation mechanism: Providers, Plugins etc.
    • Changes to any other core concepts
  2. Migration tools:
    • “Am I ready” to migrate to Airflow 2.0?
    • Upgrade configurations, etc. from 1.xx to 2.0
  3. Manual changes to migrate / upgrade
    • Migrating from “experimental” to new API
  4. Testing
  5. Docs
    • Updating.md guide especially needs to cover what the migration script does and more comprehensive and easily readable
    • Make docs-site cleaner and more organized

High-level milestones forward:

At a high level, here the key steps which need to be done: 

  1. Agree on scope (WIP)
  2. Identify the depth of completeness of the high-level items (WIP)
  3. Identify what all needs to be get done for both functional and non-functional scope
  4. Functionally complete
  5. Non-functional scope complete
  6. Validation complete
  7. Beta release of Airflow 2.0
  8. Production release of Airflow 2.0

Process:

  1. Fix a date when we create a v2-0-test branch from Airflow Master (maybe during one of the Airflow 2.0 Dev calls).
  2. After we fix the scope, we would only cherrypick commits from Airflow Master to v2-0-test branch if they are “in-scope”. Normal development would continue on Master branch i.e. PRs would be created against Airflow Master.
  3. Beta snapshots would be published to the Airflow Community to test and create issues to make sure Airflow is functioning and backwards compatible. 
  4. All the issues from 2.0.0beta1 would be fixed and 2.0.0beta2 would be published. Step (3) will be repeated.
  5. 2.0.0rc1 would then be created and an official vote would be started to release 2.0.0.
  6. If there are any bugs discovered in 2.0.0rc1, they will be fixed and a new release candidate will be published. 
    7. Step 5 and 6 will be repeated until the Vote passes

Github Issues:

To track the progress of work, we use:

...