You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 11 Next »

The purpose of this document is to capture all the key elements which need to be done in order to release Airflow 2.0 to the world.

Functional scope: 

We need to establish the scope of Airflow 2.0 so that it can be worked towards and released. The main functional elements which have been referenced and talked about by the Airflow PMC in various presentation as being components of Airflow 2.0 are:

  1. Scheduler HA 
  2. Airflow REST API
  3. DAG Versioning
  4. Functional DAGs
  5. Production-ready Docker Image
  6. Production-ready Helm Chart
    1. KEDA (with system testing)
  7. Providers Package


Approach

In the interest of clarity, the core approach is detailed below:

  1. Semantic versioning
    Since Airflow has adopted Semantic versioning, this is the opportunity to make significant changes to Airflow, including deprecating functionality and breaking changes. 

  2. Deprecate, don’t break 
    However, the key principle should be to deprecate functionality but not to break existing functionality. 

  3. Backwards compatibility 
    As far as possible, this release should be backwards compatible. If it is not possible to be backwards compatible, this should be flagged and ideally automatically migrated through a utility (which could be an add-on). 


Non-functional scope: 

We also need to establish the non-functional elements needed for a “major release” such as this. These include:

  1. Capturing changes which are needed:
    • Schedule (Setting to add choice of schedule at end or schedule at start of interval)
    • Connections (making in unique)
    • etc
    • Changes to be made to the configuration file
    • Changes to be made to DAGs / SubDags ?
    • Changes to the installation mechanism: Providers, Plugins etc.
    • Changes to any other core concepts
  2. Migration tools:
    • “Am I ready” to migrate to Airflow 2.0?
    • Upgrade configurations, etc. from 1.xx to 2.0
  3. Manual changes to migrate / upgrade
    • Migrating from “experimental” to new API
  4. Testing
  5. Docs
    • Updating.md guide especially needs to cover what the migration script does and more comprehensive and easily readable
    • Make docs-site cleaner and more organized

High-level milestones forward:

At a high level, here the key steps which need to be done: 

  1. Agree on scope
  2. Identify the depth of completeness of the high-level items (Current status)
  3. Identify what all needs to be get done for both functional and non-functional scope
  4. Functionally complete
  5. Non-functional scope complete
  6. Validation complete
  7. Beta release of Airflow 2.0
  8. Production release of Airflow 2.0


Process:

  1. Fix a date when we create a v2-0-test branch from Airflow Master (maybe during one of the Airflow 2.0 Dev calls).
  2. After we fix the scope, we would only cherrypick commits from Airflow Master to v2-0-test branch if they are “in-scope”. Normal development would continue on Master branch i.e. PRs would be created against Airflow Master.
  3. Beta snapshots would be published to the Airflow Community to test and create issues to make sure Airflow is functioning and backwards compatible. 
  4. All the issues from 2.0.0beta1 would be fixed and 2.0.0beta2 would be published. Step (3) will be repeated.
  5. 2.0.0rc1 would then be created and an official vote would be started to release 2.0.0.
  6. If there are any bugs discovered in 2.0.0rc1, they will be fixed and a new release candidate will be published. 
  7. Step 5 and 6 will be repeated until the Vote passes

Github Issues:

To track the progress of work, we use:

Dev Calls

To agree, stay aligned to the process and track the progress of Airflow 2.0 we should have regular calls between Airflow Develops (PMC Members, Committers and whoever is willing to help in Airflow 2.0).

Calendar: 


  • No labels