Versions Compared

Old Version 22

changes.mady.by.user Unknown User (vikramkoka)

Saved on

compared with

New Version 23

changes.mady.by.user Unknown User (kaxilnaik)

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  • API
    • Progress:
      • Project Board: https://github.com/apache/airflow/projects/1
        • The issues labelled with "Enhancement" are not a requirement for 2.0
      • Endpoints:
      • Permissions Model:
        • PR has been merged.
        • The next piece of work to be done is migrating existing Views to use resource-based permissions. (Github issue). This is mainly for standardizing the permissions model across API and UI.
  • Providers
    • Vote on AIP-8 took place on the mailing list.
    • There is an ongoing discussion on the same thread about SemVer vs CalVer for the Providers package.
      • The people involved on the call were leaning towards SemVer to make a clear distinction about a breaking release. This will potentially increase the work on release managers but some automation around releasing (similar to backport providers) and automation around the generation of the changelog for the providers would make the effort less painful.
    • Version Per Provide: Each Providers package would have a separate versioning i.e. we might release "google-providers 3.1" and "amazon-providers" 3.7 at the same time but the versioning for a particular provider will be independent of other providers.
  • DEV
  • SubDag Deprecation
    • There is a mailing list thread on whether or not we want to deprecate SubDags in favor of Taskgroups, the majority on the call agreed that we should not deprecate the Subdags yet and wait till people have used TaskGroups and it has feature parity with SubDags.
    • However, we should clearly recommend using TaskGroups compared to SubDags in our docs and state limitations of the SubDags.
  • Helm Chart Release
    • Deferred until 2.0 is out
    • Will be available to use from the source code of Airflow on Github but the first official release of the Helm chart will only happen after Airflow 2.0
  • Docs
    • Mailing list thread to get some feedback has been created and cross-posted across Slack and Twitter. Once we have enough feedback, Kaxil will create Github issues for them so that anyone willing to help on it can start working on it.
    • A separate section for Upgrading to 2.0 would be ideal, can be a duplicate of Updating.md but with a better structure and more organized.
  • UI Changes
  • Scheduler HA
    • Reminder: A draft PR for Scheduler HA is available for review. It would be good to get some more feedback from the wider community with their own DEV setup if possible.
  • Process
    • Any new PRs would continue to be merged until we complete the items for 2.0 and release alphas.
  • NOTE: The Timeline shown on the Planning page will be revisited every week on the Dev Call and updated if needed based on the progress towards the major features of Airflow 2.0

Things to Discuss Next

  • 28 September
    • TBD


#6: 28 Sep 2020

Attendees

NameCompany
Kaxil NaikAstronomer
Ash-Berlin Taylor

Astronomer

Vikram KokaAstronomer
Jarek PotiukPolidea
Daniel ImbermanAstronomer
Ry WalkerAstronomer
QP HouScribd
James TimminsAstronomer
Xiaodong DENG
Prakshal Jain

...

  • Scheduler HA

    • Progress

      • MySQL 8 tests have been added
      • Adoption of orphaned tasks have been added to Kubernetes and to the new CeleryKubernetesExecutor (PR)
      • All (but one) Tests are passing
      • Instructions to run the PR with Multiple Schedulers using docker-compose have been added (link to the comment)
      • Received some reviews on Monday -- Should be addressed by today/tomorrow
    • All Big/Critical Concerns have been addressed (escape hatch for locks, MySQL 8 tests, Moving callbacks to DagFileProcessor)
    • Room to improve code readability which isn't a blocker and would be addressed in follow-up PR
    • Ash is in talks with Kevin to test his Giant DAG to verify Scheduler Performance to address one of Kevin Yang's concern
    • There would be a separate PR for retrying transactions to avoid deadlocks as suggested by MySQL guide on handling deadlocks
  • API
    • Kamil and Omair are hoping to complete work on the Task Instance Endpoint PR by the end of this week
    • James is working to map existing permissions to the UI, hoping to complete by end of the week too. Link to PRs
  • Backport Providers
    • Backport packages 2020.10.5 will be released without cncf.kubernetes. We will create a separate release for cncf.kubernetes that will address the issues Kamil found in RCs. Link to VOTE thread.
  • Airflow 1.10.13 Release
  • Airflow 2.0 Alpha
    • Revisit on Thursday based on the progress with Scheduler HA & API whether we would be able to release first alpha this Friday.
  •  Others:
    • Kaxil to create Github Issues for the feedback received on the Documentation Improvement Thread and create threads asking for VOTES for previously discussed items like removing pickles.
    • Vikram will write up, share and all can review on what we mean by ALPHA and BETAs.


#8: 12 Oct 2020

Attendees

NameCompany
Kaxil NaikAstronomer
Ash-Berlin Taylor

Astronomer

Vikram KokaAstronomer
Jarek PotiukPolidea
Daniel ImbermanAstronomer
Ry WalkerAstronomer
James TimminsAstronomer
Tomasz UrbaszekPolidea
Sumit MaheshwariTwitter
QP Hou


Key Decisions

...

  • Providers 
    • Need to split each release into a separate PyPI package
    • Possibly not needed for alpha2, possibly for beta
    • Also need to split documentation for 2.0
    • Need to figure out updating and maintaining versions across providers
    • Not needed immediately, but needed for weeks to come
    • Provider versioning strategy to be documented especially for contributors and committers for the future - should be automated in the future
  • Alpha build / testing status
    • Great to see significant engagement from the community in 2.0 testing
    • Vikram reported that alpha testing going well, but still in progress
    • Alpha2 build: Targeting a second alpha build by the end of this week
    • Beta1 build pushed to next week to allow more time testing with alpha builds
    • Google cloud composer team - will start testing with beta1
  • API
    • From the API permissions standpoint, all of the changes are made and merged. 
    • Kamil made updates to the task instance endpoint.
    • API is feature complete
  • Scheduler HA
    • Jarek reported that they saw a deadlock once with MySQL 5.7 and added logging to see if it shows up again. Not a blocker since it is possible to disable locking on MySQL 5.7, but wanted to know if anyone else sees this.
    • Ping and Kevin testing Scheduler HA with a very large DAG with 21,000 tasks to benchmark in the AirBnb environment. Ash, Kevin, Ping to discuss offline. 
  • Smart Sensors
    • Yingbao commented that it was being used in AirBnb for almost a year, very stable. 
    • Ry suggested that a demo should be created for 2.0 production release to explain the capability
  • Upgrade script to 2.0 / doc
    • Daniel working on upgrade doc “Upgrading from 1.10.x to 2.0”
    • We need to check to see what all can we add to the upgrade check scripts to make it easier to migrate to 2.0
    • Database migration tests - need to be validated as well, before 2.0 RC, including the performance of the migration to 2.0. 
  • Kubernetes Executor
    • Not much to report there. 
    • Already talking about the KubernetesCelery Executor. Need to figure out if any additional testing is needed.
  • UI / UX improvements
    • Very positive feedback on the UX and the auto-refresh button 
    • Ryan Hamilton is also making accessibility-related changes to the UI.


#10: 26 Oct 2020

Attendees

Name

Company

Kaxil NaikAstronomer

Vikram Koka

Astronomer

James Timmins

Astronomer

Ry Walker

Astronomer

Jarek Potiuk

Polidea

Daniel Imberman

Astronomer

Rafal

Google

XD Deng


QP Hou

Scribd

Tomasz UrbaszekPolidea
Cindy Rogers

Discussion and Key Decisions

  • Alpha2
    • Released on 26 Oct along with the providers too
  • Providers 
    • Vote in process for Backport Providers 2020.10.29 - VOTE THREAD
    • Pending items:
      • Extra link registration of Providers to core Airflow
      • Docs
        • Split documentation from Core Airflow Docs
        • Installation of Providers
        • FAQ Section
          • Upgrading apache-airflow and providers package
          • Why do Providers have separate versioning than apache-airflow core?
          • Why extras to install providers? And what version of providers are installed when installing providers via Extras
  • KubernetesExecutor Update:
    • Draft PR to allow passing pod_template_file for overriding Pod Specification: PR Link
  • Airflow 2.0 Updating Guide
    • https://github.com/apache/airflow/blob/master/UPGRADING_TO_2.0.md contains the information required to upgrade to 2.0 from 1.10.x. This is a WIP doc and the idea is that this place should be a single place for Users to look at when they want to Upgrade to 2.0 so it should contain all the details in a digestible manner. If anyone has any suggestions on this document please create a Github issue or bring it to the mailing list.
    • ToDo:
      • Add a dedicated step to install the latest backport-providers before Upgrade.
      • Details around how long 1.10.x & backport providers will be supported should be added to Upgrading doc – Github Issue Link
  • 2.0 Beta1
    • We will wait until Friday to see how we are progressing i.e. if all the bugs found in alpha2 are fixed or not: https://github.com/apache/airflow/milestone/10. 
    • If we are not done by Friday, we will move there beta1 release to next week
  • Airflow 1.10.13

Agenda for Next Call

  • Airflow 1.10.13
  • 2.0 Beta if we don't release it this Friday
    • Separate beta
      • One to test all the features
      • One to test upgrade from 1.10.x