The tentative date for Airflow 2.0 Beta: 1st Week of October 2020 (can be revised based on the progress in the upcoming weeks)
Following Functional items were unanimously agreed that they should be a part of Airflow 2.0 and if need be Airflow 2.0 can be delayed by a few weeks if these items aren't complete:
Airflow REST API
Functional DAGs
Most of the proposal is already merged. The only pending piece is "@dag" decorator
Production-ready Docker Image
Already in good shape. Would be good to add docker-compose files and docs about using them in Quick Start guide.
Production-ready Helm chart (with KEDA)
We need a separate call to go into more details on the issues with Sources. (GH Issue and some discussion is here)
Providers Packages
A separate call to discuss some open questions on implications of not having providers directory from Airflow 2.0
Some of the issues that were mentioned were Dependency handling when new providers packages are released and its compatibility.
Scheduler HA
No Open-source PRs have been opened yet but significant progress has been made on this by Ash.
DAG Versioning will not be a part of Airflow 2.0 (and would be deferred for now) since the Scope has increased significantly after the proposal of changing the execution behaviour using possibly DAG Fetcher / DAG Manifest.
Airflow will strictly follow Semantic Versioning from Airflow 2.0. Backwards compatibility will be preserved in Airflow 2.0 where-ever possible based on the Approach described here.
Following non-functional items were discussed:
Docs:
Docs can be better organized for Airflow 2.0. We will have a separate call to discuss the exact details on what needs to be changed/re-organized.
Add missing documentation where-ever needed. Create Github issues which will allow new contributors to take on that work-load.
Schedule Interval / Execution at Start of Schedule or End of the Schedule
Deferring this to Post 2.0 as we don't have a clear agreement on this. But we all agree that adding a config to decide the edge is not a good solution as it would cause more confusion when this config is changed.
SubDAGs:
We would like to merge all the improvements to SubDAGs to make it a first-class citizen and implementing AIP-34. Based on current discussion the consensus is to make it UI-only feature introducing the concept of TaskGroup. Mailing List Thread Link
We have not come to a conclusion yet on whether this should be included in 2.0 or not. The majority is towards adding it in 2.0 (as it supports Airflow 2.0's Scalability story) and marking it as experimental.
There were some questions raised around supporting this new feature. So we decided that everyone would take a look at the PR itself and we will spend a few minutes in the next meeting to decide whether it is 2.0 or not
Simplification of KubernetesExecutor / KubernetesPodOperator
Users bash script won’t be included but anything in the core Airflow would be covered
DAG Definitions:
Changes in Path for contrib to Providers packages
DAG Interfaces: changes in arguments of a DAG / BaseOperator
Configurations:
Option to auto-replace deprecated configs with new options
Run-time Core items:
Changes like "Connection type can't be null". The upgrade-check should at least shown warning if it can't provide option to detect the type.
CLI refactor is out-of-scope
Automatic refactor is out-of-scope as it is too difficult to cover all the cases in the Users bash scripts.
This will be covered by docs or by showing warnings via the upgrade-check command
Experimental API to New API refactor is out-of-scope (will be covered by Migration docs)
We agreed that the airflow upgrade-check command needs to be available in the last release before Airflow 2.0 (1.10.x or 1.11.x)
Potential problems with time-consuming DB Migration were also discussed. If we identify such a DB Migration (example the one involving TaskInstance table) should be noted separately in Updating.md to provide a warning to the users.
DEV Calls Feedback
We agreed on having Weekly calls from 7 September onwards
Calls will start with a 5-min reviewing the progress from the last call towards 2.0
Process
A 2.0.0-test branch will be created on
Changelog:
The current way of Changelog is OK. We don't need further categorization like Webserver, Scheduler etc.
Separate Changelog would be created for Providers Packages
We need to figure a way to tag/label PRs & Issues with correct categories. Some options that were discussed were:
Adding labels on the PRs & Issues via Bot
A field in PR template for PR authors to add, the bot would then read the field which would be used to label the PR
Add rules, for example Committers needs to add appropriate labels to the PR before merging it. We could have a scheduled Github Actions workflow that would fail if it finds PRs without labels.