Versions Compared

Old Version 11

changes.mady.by.user Unknown User (kaxilnaik)

Saved on

compared with

New Version 12

changes.mady.by.user Unknown User (kaxilnaik)

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  • Smart Sensors – in 2.0 or 2.1
    • AIP-17 | PR: https://github.com/apache/airflow/pull/5499
    • We have not come to a conclusion yet on whether this should be included in 2.0 or not. The majority is towards adding it in 2.0 (as it supports Airflow 2.0's Scalability story) and marking it as experimental.
    • There were some questions raised around supporting this new feature. So we decided that everyone would take a look at the PR itself and we will spend a few minutes in the next meeting to decide whether it is 2.0 or not
  • Simplification of KubernetesExecutor / KubernetesPodOperator
  • Airflow Upgrade Check (airflow upgrade-check) command 
    • WIP PR: PR: https://github.com/apache/airflow/pull/9467 | Design Doc: https://docs.google.com/document/d/17tB9KZrH871q3AEafqR_i2I7Nrn-OT7le_P49G65VzM/edit#heading=h.vv80w6y621gv
    • Scope:
      • Users bash script won’t be included but anything in the core Airflow would be covered

      • DAG Definitions:

        • Changes in Path for contrib to Providers packages
        • DAG Interfaces: changes in arguments of a DAG / BaseOperator
      • Configurations:
        • Option to auto-replace deprecated configs with new options
      • Run-time Core items:
        • Changes like "Connection type can't be null". The upgrade-check should at least shown warning if it can't provide option to detect the type.
      • CLI refactor is out-of-scope
        • Automatic refactor is out-of-scope as it is too difficult to cover all the cases in the Users bash scripts.
        • This will be covered by docs or by showing warnings via the upgrade-check command
      • Experimental API to New API refactor is out-of-scope (will be covered by Migration docs)
    • We agreed that the airflow upgrade-check command needs to be available in the last release before Airflow 2.0 (1.10.x or 1.11.x)
  • Potential problems with time-consuming DB Migration were also discussed. If we identify such a DB Migration (example the one involving TaskInstance table) should be noted separately in Updating.md to provide a warning to the users.
  • DEV Calls Feedback
    • We agreed on having Weekly calls from 7 September onwards
    • Calls will start with a 5-min reviewing the progress from the last call towards 2.0
  • Process
    • A 2.0.0-test branch will be created on  
    • Changelog:
      • The current way of Changelog is OK. We don't need further categorization like Webserver, Scheduler etc.
      • Separate Changelog would be created for Providers Packages
      • We need to figure a way to tag/label PRs & Issues with correct categories. Some options that were discussed were:
        • Adding labels on the PRs & Issues via Bot
        • A field in PR template for PR authors to add, the bot would then read the field which would be used to label the PR
        • Add rules, for example Committers needs to add appropriate labels to the PR before merging it. We could have a scheduled Github Actions workflow that would fail if it finds PRs without labels.


#3: 7 Sep 2020

Attendees

NameCompany
Kaxil NaikAstronomer
Ash-Berlin TaylorAstronomer
Tomek UrbaszekPolidea
Kevin YangAirbnb
Jarek PotiukPolidea
Daniel ImbermanAstronomer
Vikram KokaAstronomer

Xiaodong DENG

-
Ry WalkerAstronomer
Shekhar SinghGojek
Rafal BiegaczGoogle
Anita FronczakGoogle

Key Decisions

  • Smart Sensors
    • Will be included in 2.0 as an early-access feature with a clear note that this feature might potentially change in future Airflow version with breaking changes.
    • Airbnb team would be happy to help on the support side answering questions related to Smart Sensor.
    • Add docs around different execution modes for Sensor: Poke mode, Reschedule mode vs Smart Sensor
  • Providers Packages
    • We had a consensus on releasing providers packages separately mainly because of the following reasons:
      • Separate cadence for providers compared to Airflow, so bugs in operator/hooks can be fixed lot faster.
      • Enterprises generally would not like to upgrade the “core” (Scheduler) as a small bug can break the deployment and affect all the DAGs
      • Breaking library changes (new version of a library) can be fixed with a new version of Backport/Providers
      • Upgrades of backport providers are not “that” destructive i.e. even if you upgrade to a newer version and find a bug, you could go back to the previous version without causing any issues at all.
    • Open questions / Action Items:
      • How would users figure out “breaking changes” with CALVER Versioning (which is very clear with SEMVER)?
      • Use plugin Mechanism to:
        • Register Connections from an external provider to allow custom field or hide existing form fields.
        • Register Operator Extra links for operators in providers so that a change is not required in Airflow
    • Backport providers will only be supported/released for three months after 2.0.0 released
  • Timeline to Airflow 2.0
    • Only critical fixes (fixes to bugs that takedown Production system) will be backported to 1.10.x core for a further three month after Airflow 2.0 is released.


Date

Milestone

Week of 7 Sep 2020Create the 2.0.0-test branch

While the scope is fluid, we would be rebasing this test branch from master. After we completely freeze the scope, we would only cherrypick commits from Airflow Master to v2-0-test branch if they are “in-scope”. Normal development would continue on Master branch i.e. PRs would be created against Airflow Master.

Week of 28 Sep 2020Cut Functionally complete 2.0 alpha release
Week of 12 Oct 2020Cut first 2.0 beta release

Beta snapshots would be published to the Airflow Community to test and create issues to make sure Airflow is functioning and backwards compatible. 

Week of 19 Oct 2020Cut bridge release based on 1.10.x - jump-off point to 2.0. Probably 1.10.13 or 1.10.14 containing upgrade check scripts for 2.0
Week of 26 Oct 2020Cut second 2.0 beta release
Week of 9 Nov 2020Cut third 2.0 beta release
Week of 23 Nov 2020Cut first 2.0 release candidate (2.0.0rc1)

Things to Discuss Next

...