Meeting Notes

#1: 10 Aug 2020 - Planning & Scoping

Attendees

Key Decisions

• The tentative date for Airflow 2.0 Beta: 1st Week of October 2020 (can be revised based on the progress in the upcoming weeks)
• Following Functional items were unanimously agreed that they should be a part of Airflow 2.0 and if need be Airflow 2.0 can be delayed by a few weeks if these items aren't complete:
  • Airflow REST API 
  • Functional DAGs
    • Most of the proposal is already merged. The only pending piece is "@dag" decorator
  • Production-ready Docker Image
    • Already in good shape. Would be good to add docker-compose files and docs about using them in Quick Start guide.
  • Production-ready Helm chart (with KEDA)
    • We need a separate call to go into more details on the issues with Sources. (GH Issue and some discussion is here)
  • Providers Packages
    • A separate call to discuss some open questions on implications of not having providers directory from Airflow 2.0
    • Some of the issues that were mentioned were Dependency handling when new providers packages are released and its compatibility.
  • Scheduler HA
    • No Open-source PRs have been opened yet but significant progress has been made on this by Ash.
• DAG Versioning will not be a part of Airflow 2.0 (and would be deferred for now) since the Scope has increased significantly after the proposal of changing the execution behaviour using possibly DAG Fetcher / DAG Manifest.
• Airflow will strictly follow Semantic Versioning from Airflow 2.0. Backwards compatibility will be preserved in Airflow 2.0 where-ever possible based on the Approach described here.
• Following non-functional items were discussed:
  • Docs:
    • Docs can be better organized for Airflow 2.0. We will have a separate call to discuss the exact details on what needs to be changed/re-organized.
    • Add missing documentation where-ever needed. Create Github issues which will allow new contributors to take on that work-load.
  • Schedule Interval / Execution at Start of Schedule or End of the Schedule 
    • Deferring this to Post 2.0 as we don't have a clear agreement on this. But we all agree that adding a config to decide the edge is not a good solution as it would cause more confusion when this config is changed.
  • SubDAGs:
    • We would like to merge all the improvements to SubDAGs to make it a first-class citizen and implementing AIP-34. Based on current discussion the consensus is to make it UI-only feature introducing the concept of TaskGroup. Mailing List Thread Link

#2: 24 Aug 2020

Attendees

Key Decisions

• Smart Sensors – in 2.0 or 2.1
  • AIP-17 | PR: https://github.com/apache/airflow/pull/5499
  • We have not come to a conclusion yet on whether this should be included in 2.0 or not. The majority is towards adding it in 2.0 (as it supports Airflow 2.0's Scalability story) and marking it as experimental.
  • There were some questions raised around supporting this new feature. So we decided that everyone would take a look at the PR itself and we will spend a few minutes in the next meeting to decide whether it is 2.0 or not
• Simplification of KubernetesExecutor / KubernetesPodOperator
  • PR: https://github.com/apache/airflow/pull/10393
  • This will be part of Airflow 2.0
• Airflow Upgrade Check (airflow upgrade-check) command 
  • WIP PR: PR: https://github.com/apache/airflow/pull/9467 | Design Doc: https://docs.google.com/document/d/17tB9KZrH871q3AEafqR_i2I7Nrn-OT7le_P49G65VzM/edit#heading=h.vv80w6y621gv
  • Scope:
    • Users bash script won’t be included but anything in the core Airflow would be covered

    • DAG Definitions:

      • Changes in Path for contrib to Providers packages
      • DAG Interfaces: changes in arguments of a DAG / BaseOperator
    • Configurations:
      • Option to auto-replace deprecated configs with new options
    • Run-time Core items:
      • Changes like "Connection type can't be null". The upgrade-check should at least shown warning if it can't provide option to detect the type.
    • CLI refactor is out-of-scope
      • Automatic refactor is out-of-scope as it is too difficult to cover all the cases in the Users bash scripts.
      • This will be covered by docs or by showing warnings via the upgrade-check command
    • Experimental API to New API refactor is out-of-scope (will be covered by Migration docs)
  • We agreed that the airflow upgrade-check command needs to be available in the last release before Airflow 2.0 (1.10.x or 1.11.x)
• Potential problems with time-consuming DB Migration were also discussed. If we identify such a DB Migration (example the one involving TaskInstance table) should be noted separately in Updating.md to provide a warning to the users.
• DEV Calls Feedback
  • We agreed on having Weekly calls from 7 September onwards
  • Calls will start with a 5-min reviewing the progress from the last call towards 2.0
• Process
  • A 2.0.0-test branch will be created on 10 Sep 2020 
  • Changelog:
    • The current way of Changelog is OK. We don't need further categorization like Webserver, Scheduler etc.
    • Separate Changelog would be created for Providers Packages
    • We need to figure a way to tag/label PRs & Issues with correct categories. Some options that were discussed were:
      • Adding labels on the PRs & Issues via Bot
      • A field in PR template for PR authors to add, the bot would then read the field which would be used to label the PR
      • Add rules, for example Committers needs to add appropriate labels to the PR before merging it. We could have a scheduled Github Actions workflow that would fail if it finds PRs without labels.

#3: 7 Sep 2020

Attendees

Key Decisions

• Smart Sensors
  • Will be included in 2.0 as an early-access feature with a clear note that this feature might potentially change in future Airflow version with breaking changes.
  • Airbnb team would be happy to help on the support side answering questions related to Smart Sensor.
  • Add docs around different execution modes for Sensor: Poke mode, Reschedule mode vs Smart Sensor
• Providers Packages
  • We had a consensus on releasing providers packages separately mainly because of the following reasons:
    • Separate cadence for providers compared to Airflow, so bugs in operator/hooks can be fixed lot faster.
    • Enterprises generally would not like to upgrade the “core” (Scheduler) as a small bug can break the deployment and affect all the DAGs
    • Breaking library changes (new version of a library) can be fixed with a new version of Backport/Providers
    • Upgrades of backport providers are not “that” destructive i.e. even if you upgrade to a newer version and find a bug, you could go back to the previous version without causing any issues at all.
  • Open questions / Action Items:
  • How would users figure out “breaking changes” with CALVER Versioning (which is very clear with SEMVER)?
  • Use plugin Mechanism to:
    • Register Connections from an external provider to allow custom field or hide existing form fields.
    • Register Operator Extra links for operators in providers so that a change is not required in Airflow
  • Backport providers will only be supported/released for three months after 2.0.0 released
• Timeline to Airflow 2.0
  • Only critical fixes (fixes to bugs that takedown Production system) will be backported to 1.10.x core for six months after Airflow 2.0 is released.

#4: 14 Sep 2020

Attendees

Key Decisions

• Updates
  • Airflow v2-0-test branch has already been cut and currently manually rebased on top of the Master. Currently, we don't run CI as the branch is in-sync with Master. As soon as we have a PR / commit that we don't want to have it in 2.0 we will diverge v2-0-test branch from Master and start running tests against it.
  • The upgrade-check PR was merged, we now need to define more rules to add more checks.
• API
  • Progress:
    • Project Board: https://github.com/apache/airflow/projects/1
      • The issues labelled with "Enhancement" are not a requirement for 2.0
    • Endpoints:
      • Task Instance Endpoint is WIP, all the other endpoints have been implemented.
    • Permissions Model:
      • On-going discussion on the PR but close to completion.
      • The next piece of work to be done is migrating existing Views to use resource-based permissions. (Github issue). This is mainly for standardizing the permissions model across API and UI.
• Improvements to SubDags / Concept of TaskGroup
  • AIP-34 | PR introduced the concepts of TaskGroup and will be included in Airflow 2.0.
    • The PR implements TaskGroups for Graph View, the Tree View will be implemented in follow-up PRs.
  • Follow-up items from the discussion:
    • Discuss on mailing list whether we should deprecate SubDags in favour of TaskGroup in 2.0 or wait until Airflow 2.1 or 2.2
    • Add docs around when to use TaskGroup vs SubDag and potentially listing PROs and CONS.
• Scheduler HA (AIP-15 )
  • A Draft PR has been created to enable code reviews and to allow the members of the community to start testing it with various setups. 
  • To get the most benefit of Scheduler HA on MySQL, users will need to use MySQL 8. This is because MySQL 5.7 does not support SKIP LOCK feature but note that MySQL 5.7 will still continue to work with at least the same or improved performance as now.
  • Astronomer has done performance testing with different Scenarios and will publish benchmarks over the coming weeks. Google Composer Team + Polidea said that they would be happy to carry out various tests for Scheduler HA as well.
  • There were some concerns raised around LOCKING Timeout periods and the usage of DAG Serialization. More testing in the upcoming weeks should help mitigate any concerns and help fix the bugs if discovered.
  • Docs:
    • Explicitly mention that for HA Scheduler reads some of the properties from serialized_dag table. Users can turn on/off DAG Serialization in the Webserver but the Scheduler will continue using it.
    • Do we recommend 2 schedulers for Production deployments?
    • X Schedulers vs single Scheduler. Use case when one would be better than the other.
      • Some kind of Bell Curve showing an increase in Schedulers stops improving performance and maybe also degrades. This is intended to give guidance around what number of schedulers to run based on expected load, since this decision could be based on multiple factors.
  • Follow up items:
    • Create mailing list thread to discuss "Removing Pickling from Airflow 2.0". Currently, pickled dags are only supported by CeleryExecutor and we have a flag on airflow scheduler (--do-pickle) and "--ship-dag" on airflow tasks run command. If we want to remove pickling Airflow 2.0 is the right time or we shouldn't do it until 3.0
• Helm Chart
  • We will continue focusing on getting Airflow 2.0 out so the first official release of Helm Chart might need to wait.
  • The issue with Helm Chart sources was fixed and there are no blockers currently if we were to release it at some point in the near future.
  • Enhancements (but not blockers) are:
    • Better Test Coverage with integration tests
    • Docs pointing to the chart on the Airflow Website or the docsite
  • The artifacts for the Helm chart would be published at https://downloads.apache.org/airflow/
  • There is still an open question around Helm Chart Versioning Policy i.e. do we want to tie-in Airflow Versions with Helm Chart? Or do we just start from 1.0.0?  This needs to be decided before the release of the Helm Chart.

#5: 21 Sep 2020

Attendees

Key Decisions

• Updates
  • Airflow v2-0-test branch has already been cut and currently manually rebased on top of the Master. Currently, we don't run CI as the branch is in-sync with Master. As soon as we have a PR / commit that we don't want to have it in 2.0 we will diverge v2-0-test branch from Master and start running tests against it.
  • The upgrade-check PR was merged, we now need to define more rules to add more checks.
• API
  • Progress:
    • Project Board: https://github.com/apache/airflow/projects/1
      • The issues labelled with "Enhancement" are not a requirement for 2.0
    • Endpoints:
      • Task Instance Endpoint is WIP, all the other endpoints have been implemented.
    • Permissions Model:
      • PR has been merged.
      • The next piece of work to be done is migrating existing Views to use resource-based permissions. (Github issue). This is mainly for standardizing the permissions model across API and UI.
• Providers
  • Vote on AIP-8 took place on the mailing list.

Things to Discuss Next

• 28 September (TBD)
  • Finish up open discussion items from the earlier meeting if not yet resolved:
    • Providers versioning,
    • SubDag deprecation,
    • Helm Chart release,
    • REST API permissions 
    • Docs changes
  • UI Changes for 2.0
    • Minimum effort changes: CSS/colours/spacing to make the UI look a bit modern
  • Process:
    • When should we defer the in-scope items to post-2.0
      • Completion by a date?
      • Progress by a date?