Versions Compared

Old Version 5

changes.mady.by.user Unknown User (vikramkoka)

Saved on

compared with

New Version 6

changes.mady.by.user Kamil Bregula

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

NameCompay
Kaxil NaikAstronomer
Tomek UrbaszekPolidea
James TimminsAstronomer
Jarek PotiukPolidea
Tobiasz KędzierskiPolidea
Anita FronczakGoogle
Goran ObradovicGoogle
Mateusz HencGoogle
Rafal BiegaczGoogle
Daniel ImbermanAstronomer
Vikram KokaAstronomer
QP HouScribd
Ephraim Anierobi -
Felix UellendallDigitas Pixelpark
Shekhar Singh

Gojek

Kamill BregułaPolidea


Key Decisions

  • The tentative date for Airflow 2.0 Beta: 1st Week of October 2020 (can be revised based on the progress in the upcoming weeks)
  • Following Functional items were unanimously agreed that they should be a part of Airflow 2.0 and if need be Airflow 2.0 can be delayed by a few weeks if these items aren't complete:
    • Airflow REST API 
    • Functional DAGs
      • Most of the proposal is already merged. The only pending piece is "@dag" decorator
    • Production-ready Docker Image
      • Already in good shape. Would be good to add docker-compose files and docs about using them in Quick Start guide.
    • Production-ready Helm chart (with KEDA)
      • We need a separate call to go into more details on the issues with Sources. (GH Issue and some discussion is here)
    • Providers Packages
      • A separate call to discuss some open questions on implications of not having providers directory from Airflow 2.0
      • Some of the issues that were mentioned were Dependency handling when new providers packages are released and its compatibility.
    • Scheduler HA
      • No Open-source PRs have been opened yet but significant progress has been made on this by Ash.
  • DAG Versioning will not be a part of Airflow 2.0 (and would be deferred for now) since the Scope has increased significantly after the proposal of changing the execution behaviour using possibly DAG Fetcher / DAG Manifest.
  • Airflow will strictly follow Semantic Versioning from Airflow 2.0. Backwards compatibility will be preserved in Airflow 2.0 where-ever possible based on the Approach described here.
  • Following non-functional items were discussed:
    • Docs:
      • Docs can be better organized for Airflow 2.0. We will have a separate call to discuss the exact details on what needs to be changed/re-organized.
      • Add missing documentation where-ever needed. Create Github issues which will allow new contributors to take on that work-load.
    • Schedule Interval / Execution at Start of Schedule or End of the Schedule 
      • Deferring this to Post 2.0 as we don't have a clear agreement on this. But we all agree that adding a config to decide the edge is not a good solution as it would cause more confusion when this config is changed.
    • SubDAGs:
      • We would like to merge all the improvements to SubDAGs to make it a first-class citizen and implementing AIP-34. Based on current discussion the consensus is to make it UI-only feature introducing the concept of TaskGroup. Mailing List Thread Link

...