Page History

...

• API
  • Task Instance Endpoint: Only thing open. Kaxil to speak with Kamil to get status. 
  • Existing permissions map to UI: WIP PRs. Feedback requested by James on this PR https://github.com/apache/airflow/pull/11158/
  • Migrations: Is there a different migration process needed? Can be run as a standard alembic migration. 
  • Clients: QP said that the GO client based on the REST API is already completed and he is using it as part of his airflow-terraform module.
• UI Improvements
  • New HomePage: Ryan has updated the UX (look) and shared on slack. Split Actions and Buttons, using Google Material Fonts. Feedback from meeting was very positive. PR to be final tomorrow. 
• Splitting Providers Package
  • Separate provider packages from core: Need some help to get this wrapped up. https://github.com/apache/airflow/issues/11163
  • Build optimization: This would really help speed-up builds by only running tests for changed providers https://github.com/apache/airflow/issues/10507
  • SemVer: Kaxil to send email to dev list confirming the decision in the meeting about using SemVer (lazy-consensus)
• Backport Providers in next 1.10.x release
  • EmailOperator: Location has changed (moved to core) 
  • CNCF.Kubernetes providers: Need to be checked if these can be released. Insert github issue link created by Kamil for core operators
• Timeline update:
  • Functionally complete build: Will be on the week of Oct 5th, instead of this week as described above in the Scheduler HA notes. 

#7: 5 Oct 2020

Attendees

Key Decisions

• Scheduler HA

  • Progress

    • MySQL 8 tests have been added
    • Adoption of orphaned tasks have been added to Kubernetes and to the new CeleryKubernetesExecutor (PR)
    • All (but one) Tests are passing
    • Instructions to run the PR with Multiple Schedulers using docker-compose have been added (link to the comment)
    • Received some reviews on Monday -- Should be addressed by today/tomorrow
  • All Big/Critical Concerns have been addressed (escape hatch for locks, MySQL 8 tests, Moving callbacks to DagFileProcessor)
  • Room to improve code readability which isn't a blocker and would be addressed in follow-up PR
  • Ash is in talks with Kevin to test his Giant DAG to verify Scheduler Performance to address one of Kevin Yang's concern
  • There would be a separate PR for retrying transactions to avoid deadlocks as suggested by MySQL guide on handling deadlocks
• API
  • Kamil and Omair are hoping to complete work on the Task Instance Endpoint PR by the end of this week
  • James is working to map existing permissions to the UI, hoping to complete by end of the week too. Link to PRs
• Backport Providers
  • Backport packages 2020.10.5 will be released without cncf.kubernetes. We will create a separate release for cncf.kubernetes that will address the issues Kamil found in RCs. Link to VOTE thread.
• Airflow 1.10.13 Release
  • Will be released as per the planned timeline (Airflow 2.0 - Planning) i.e. Last Week of October. 
• Airflow 2.0 Alpha
  • Revisit on Thursday based on the progress with Scheduler HA & API whether we would be able to release first alpha this Friday.
•  Others:
  • Kaxil to create Github Issues for the feedback received on the Documentation Improvement Thread and create threads asking for VOTES for previously discussed items like removing pickles.
  • Vikram will write up, share and all can review on what we mean by ALPHA and BETAs.

#8: 12 Oct 2020

Attendees

Key Decisions

• Scheduler HA

  • PR Merged

• Github Project for tracking Alpha & Beta
  • Alpha1: https://github.com/apache/airflow/milestone/9
  • Alpha2: https://github.com/apache/airflow/milestone/11
  • Beta: https://github.com/apache/airflow/milestone/10
• Vikram to send an email for Github Issues triage with other folks
• Blockers for releasing Alpha-1
  • Renaming Functional DAGs
• Docs:
  • Clearly state "Functional DAGs" (decorator and not setting dependencies) only work with PythonOperator
  • Adding docs around using “Task.output” (i.e. tasks not using PythonOpertator)

• API

  • Task Instance Endpoint
  • Draft/WIP PR: https://github.com/apache/airflow/pull/9597
  • Should be done by the end of this week
  • Existing Permission and Map them to UI (Flask View Permissions).
    • WIP PRs: https://github.com/apache/airflow/pulls/jhtimmins
    • Should be done by the end of this week

#9: 19 Oct 2020

Attendees

Discussion and Key Decisions

• Providers 
  • Need to split each release into a separate PyPI package
  • Possibly not needed for alpha2, possibly for beta
  • Also need to split documentation for 2.0
  • Need to figure out updating and maintaining versions across providers
  • Not needed immediately, but needed for weeks to come
  • Provider versioning strategy to be documented especially for contributors and committers for the future - should be automated in the future
• Alpha build / testing status
  • Great to see significant engagement from the community in 2.0 testing
  • Vikram reported that alpha testing going well, but still in progress
  • Alpha2 build: Targeting a second alpha build by the end of this week
  • Beta1 build pushed to next week to allow more time testing with alpha builds
  • Google cloud composer team - will start testing with beta1
• API
  • From the API permissions standpoint, all of the changes are made and merged. 
  • Kamil made updates to the task instance endpoint.
  • API is feature complete
• Scheduler HA
  • Jarek reported that they saw a deadlock once with MySQL 5.7 and added logging to see if it shows up again. Not a blocker since it is possible to disable locking on MySQL 5.7, but wanted to know if anyone else sees this.
  • Ping and Kevin testing Scheduler HA with a very large DAG with 21,000 tasks to benchmark in the AirBnb environment. Ash, Kevin, Ping to discuss offline. 

• Smart Sensors

• Yingbao commented that it was being used in AirBnb for almost a year, very stable. 
• Ry suggested that a demo should be created for 2.0 production release to explain the capability

• Upgrade script to 2.0 / doc

• Daniel working on upgrade doc “Upgrading from 1.10.x to 2.0”
• We need to check to see what all can we add to the upgrade check scripts to make it easier to migrate to 2.0
• Database migration tests - need to be validated as well, before 2.0 RC, including the performance of the migration to 2.0. 

• Kubernetes Executor

• Not much to report there. 
• Already talking about the KubernetesCelery Executor. Need to figure out if any additional testing is needed.

• UI / UX improvements

• Very positive feedback on the UX and the auto-refresh button 
• Ryan Hamilton is also making accessibility-related changes to the UI.