Versions Compared

Old Version 17

changes.mady.by.user Unknown User (kaxilnaik)

Saved on

compared with

New Version 18

changes.mady.by.user Unknown User (kaxilnaik)

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  • Updates
    • Airflow v2-0-test branch has already been cut and currently manually rebased on top of the Master. Currently, we don't run CI as the branch is in-sync with Master. As soon as we have a PR / commit that we don't want to have it in 2.0 we will diverge v2-0-test branch from Master and start running tests against it.
    • The upgrade-check PR was merged, we now need to define more rules to add more checks.
  • API
    • Progress:
      • Project Board: https://github.com/apache/airflow/projects/1
        • The issues labelled with "Enhancement" are not a requirement for 2.0
      • Endpoints:
      • Permissions Model:
        • On-going discussion on the PR but close to completion.
        • The next piece of work to be done is migrating existing Views to use resource-based permissions. (Github issue). This is mainly for standardizing the permissions model across API and UI.
  • Improvements to SubDags / Concept of TaskGroup
    • AIP-34 | PR introduced the concepts of TaskGroup and will be included in Airflow 2.0.
      • The PR implements TaskGroups for Graph View, the Tree View will be implemented in follow-up PRs.
    • Follow-up items from the discussion:
      • Discuss on mailing list whether we should deprecate SubDags in favour of TaskGroup in 2.0 or wait until Airflow 2.1 or 2.2
      • Add docs around when to use TaskGroup vs SubDag and potentially listing PROs and CONS.
  • Scheduler HA (AIP-15 )
    • A Draft PR has been created to enable code reviews and to allow the members of the community to start testing it with various setups. 
    • To get the most benefit of Scheduler HA on MySQL, users will need to use MySQL 8. This is because MySQL 5.7 does not support SKIP LOCK feature but note that MySQL 5.7 will still continue to work with at least the same or improved performance as now.
    • Astronomer has done performance testing with different Scenarios and will publish benchmarks over the coming weeks. Google Composer Team + Polidea said that they would be happy to carry out various tests for Scheduler HA as well.
    • There were some concerns raised around LOCKING Timeout periods and the usage of DAG Serialization. More testing in the upcoming weeks should help mitigate any concerns and help fix the bugs if discovered.
    • Docs:
      • Explicitly mention that for HA Scheduler reads some of the properties from serialized_dag table. Users can turn on/off DAG Serialization in the Webserver but the Scheduler will continue using it.
      • Do we recommend 2 schedulers for Production deployments?
      • X Schedulers vs single Scheduler. Use case when one would be better than the other.
        • Some kind of Bell Curve showing an increase in Schedulers stops improving performance and maybe also degrades. This is intended to give guidance around what number of schedulers to run based on expected load, since this decision could be based on multiple factors.
    • Follow up items:
      • Create mailing list thread to discuss "Removing Pickling from Airflow 2.0". Currently, pickled dags are only supported by CeleryExecutor and we have a flag on airflow scheduler (--do-pickle) and "--ship-dag" on airflow tasks run command. If we want to remove pickling Airflow 2.0 is the right time or we shouldn't do it until 3.0
  • Helm Chart
    • We will continue focusing on getting Airflow 2.0 out so the first official release of Helm Chart might need to wait.
    • The issue with Helm Chart sources was fixed and there are no blockers currently if we were to release it at some point in the near future.
    • Enhancements (but not blockers) are:
      • Better Test Coverage with integration tests
      • Docs pointing to the chart on the Airflow Website or the docsite
    • The artifacts for the Helm chart would be published at https://downloads.apache.org/airflow/
    • There is still an open question around Helm Chart Versioning Policy i.e. do we want to tie-in Airflow Versions with Helm Chart? Or do we just start from 1.0.0?  This needs to be decided before the release of the Helm Chart.


#5: 21 Sep 2020

Attendees

NameCompany
Kaxil NaikAstronomer
Ash-Berlin Taylor

Astronomer

Vikram KokaAstronomer
Tomek UrbaszekPolidea
Jarek PotiukPolidea
Daniel ImbermanAstronomer
Shekhar SinghGojek
Ry WalkerAstronomer
Rafal BiegaczGoogle


Key Decisions

  • Updates
    • Airflow v2-0-test branch has already been cut and currently manually rebased on top of the Master. Currently, we don't run CI as the branch is in-sync with Master. As soon as we have a PR / commit that we don't want to have it in 2.0 we will diverge v2-0-test branch from Master and start running tests against it.
    • The upgrade-check PR was merged, we now need to define more rules to add more checks.
  • API
    • Progress:
      • Project Board: https://github.com/apache/airflow/projects/1
        • The issues labelled with "Enhancement" are not a requirement for 2.0
      • Endpoints:
      • Permissions Model:
        • PR has been merged.
        • The next piece of work to be done is migrating existing Views to use resource-based permissions. (Github issue). This is mainly for standardizing the permissions model across API and UI.
  • Providers

Things to Discuss Next

    • There is an ongoing discussion on the same thread about SemVer vs CalVer for the Providers package.
      • The people involved on the call were leaning towards SemVer to make a clear distinction about a breaking release. This will potentially increase the work on release managers but some automation around releasing (similar to backport providers) and automation around the generation of the changelog for the providers would make the effort less painful.
    • Version Per Provide: Each Providers package would have a separate versioning i.e. we might release "google-providers 3.1" and "amazon-providers" 3.7 at the same time but the versioning for a particular provider will be independent of other providers.
  • DEV
  • SubDag Deprecation
    • There is a mailing list thread on whether or not we want to deprecate SubDags in favor of Taskgroups, the majority on the call agreed that we should not deprecate the Subdags yet and wait till people have used TaskGroups and it has feature parity with SubDags.
    • However, we should clearly recommend using TaskGroups compared to SubDags in our docs and state limitations of the SubDags.
  • Helm Chart Release
    • Deferred until 2.0 is out
    • Will be available to use from the source code of Airflow on Github but the first official release of the Helm chart will only happen after Airflow 2.0
  • Docs
    • Mailing list thread to get some feedback has been created and cross-posted across Slack and Twitter. Once we have enough feedback, Kaxil will create Github issues for them so that anyone willing to help on it can start working on it.
    • A separate section for Upgrading to 2.0 would be ideal, can be a duplicate of Updating.md but with a better structure and more organized.
  • UI Changes
  • Scheduler HA
    • Reminder: A draft PR for Scheduler HA is available for review. It would be good to get some more feedback from the wider community with their own DEV setup if possible.
  • Process
    • Any new PRs would continue to be merged until we complete the items for 2.0 and release alphas.
  • NOTE: The Timeline shown on the Planning page will be revisited every week on the Dev Call and updated if needed based on the progress towards the major features of Airflow 2.0


Things to Discuss Next

  • 28 September
    • TBD
    28 September (TBD)
    • Finish up open discussion items from the earlier meeting if not yet resolved:
      • Providers versioning,
      • SubDag deprecation,
      • Helm Chart release,
      • REST API permissions 
      • Docs changes
    • UI Changes for 2.0
      • Minimum effort changes: CSS/colours/spacing to make the UI look a bit modern
    • Process:When should we defer the in-scope items to post-2.0
    • Completion by a date?
      • Progress by a date?