Meeting Notes

#1: 10 Aug 2020 - Planning & Scoping

Attendees

Key Decisions

• The tentative date for Airflow 2.0 Beta: 1st Week of October 2020 (can be revised based on the progress in the upcoming weeks)
• Following Functional items were unanimously agreed that they should be a part of Airflow 2.0 and if need be Airflow 2.0 can be delayed by a few weeks if these items aren't complete:
  • Airflow REST API 
  • Functional DAGs
    • Most of the proposal is already merged. The only pending piece is "@dag" decorator
  • Production-ready Docker Image
    • Already in good shape. Would be good to add docker-compose files and docs about using them in Quick Start guide.
  • Production-ready Helm chart (with KEDA)
    • We need a separate call to go into more details on the issues with Sources. (GH Issue and some discussion is here)
  • Providers Packages
    • A separate call to discuss some open questions on implications of not having providers directory from Airflow 2.0
    • Some of the issues that were mentioned were Dependency handling when new providers packages are released and its compatibility.
  • Scheduler HA
    • No Open-source PRs have been opened yet but significant progress has been made on this by Ash.
• DAG Versioning will not be a part of Airflow 2.0 (and would be deferred for now) since the Scope has increased significantly after the proposal of changing the execution behaviour using possibly DAG Fetcher / DAG Manifest.
• Airflow will strictly follow Semantic Versioning from Airflow 2.0. Backwards compatibility will be preserved in Airflow 2.0 where-ever possible based on the Approach described here.
• Following non-functional items were discussed:
  • Docs:
    • Docs can be better organized for Airflow 2.0. We will have a separate call to discuss the exact details on what needs to be changed/re-organized.
    • Add missing documentation where-ever needed. Create Github issues which will allow new contributors to take on that work-load.
  • Schedule Interval / Execution at Start of Schedule or End of the Schedule 
    • Deferring this to Post 2.0 as we don't have a clear agreement on this. But we all agree that adding a config to decide the edge is not a good solution as it would cause more confusion when this config is changed.
  • SubDAGs:
    • We would like to merge all the improvements to SubDAGs to make it a first-class citizen and implementing AIP-34. Based on current discussion the consensus is to make it UI-only feature introducing the concept of TaskGroup. Mailing List Thread Link

Things to Discuss Next

• Process
  • Decide on a date to create a 2.0.0-test branch
  • When should we defer the in-scope items to post-2.0
    • Completion by a date?
    • Progress by a date?
  • Creating a Changelog for 2.0
  • Beta Testing
    
    • Need help from Airflow Vendors and power customers like Astronomer, Google, AirBnb
• Smart Sensors – in 2.0 or 2.1
• TBD