• Created by Unknown User (kaxilnaik), last updated by Unknown User (vikramkoka) on Aug 12, 2020 2 minute read

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 5 Next »

#1: 10 Aug 2020 - Planning & Scoping

Attendees

NameCompay
Kaxil NaikAstronomer
Tomek UrbaszekPolidea
James TimminsAstronomer
Jarek PotiukPolidea
Tobiasz KędzierskiPolidea
Anita FronczakGoogle
Goran ObradovicGoogle
Mateusz HencGoogle
Rafal BiegaczGoogle
Daniel ImbermanAstronomer
Vikram KokaAstronomer
QP HouScribd
Ephraim Anierobi -
Felix UellendallDigitas Pixelpark
Shekhar SinghGojek

Key Decisions

  • The tentative date for Airflow 2.0 Beta: 1st Week of October 2020 (can be revised based on the progress in the upcoming weeks)
  • Following Functional items were unanimously agreed that they should be a part of Airflow 2.0 and if need be Airflow 2.0 can be delayed by a few weeks if these items aren't complete:
    • Airflow REST API 
    • Functional DAGs
      • Most of the proposal is already merged. The only pending piece is "@dag" decorator
    • Production-ready Docker Image
      • Already in good shape. Would be good to add docker-compose files and docs about using them in Quick Start guide.
    • Production-ready Helm chart (with KEDA)
      • We need a separate call to go into more details on the issues with Sources. (GH Issue and some discussion is here)
    • Providers Packages
      • A separate call to discuss some open questions on implications of not having providers directory from Airflow 2.0
      • Some of the issues that were mentioned were Dependency handling when new providers packages are released and its compatibility.
    • Scheduler HA
      • No Open-source PRs have been opened yet but significant progress has been made on this by Ash.
  • DAG Versioning will not be a part of Airflow 2.0 (and would be deferred for now) since the Scope has increased significantly after the proposal of changing the execution behaviour using possibly DAG Fetcher / DAG Manifest.
  • Airflow will strictly follow Semantic Versioning from Airflow 2.0. Backwards compatibility will be preserved in Airflow 2.0 where-ever possible based on the Approach described here.
  • Following non-functional items were discussed:
    • Docs:
      • Docs can be better organized for Airflow 2.0. We will have a separate call to discuss the exact details on what needs to be changed/re-organized.
      • Add missing documentation where-ever needed. Create Github issues which will allow new contributors to take on that work-load.
    • Schedule Interval / Execution at Start of Schedule or End of the Schedule 
      • Deferring this to Post 2.0 as we don't have a clear agreement on this. But we all agree that adding a config to decide the edge is not a good solution as it would cause more confusion when this config is changed.
    • SubDAGs:
      • We would like to merge all the improvements to SubDAGs to make it a first-class citizen and implementing AIP-34. Based on current discussion the consensus is to make it UI-only feature introducing the concept of TaskGroup. Mailing List Thread Link

Things to Discuss Next

  • Process
    • Decide on a date to create a 2.0.0-test branch
    • When should we defer the in-scope items to post-2.0
      • Completion by a date?
      • Progress by a date?
    • Creating a Changelog for 2.0
    • Beta Testing
      • Need help from Airflow Vendors and power customers like Astronomer, Google, AirBnb
  • Smart Sensors – in 2.0 or 2.1
  • TBD
  • No labels