Page History

...

• 28 September
  • TBD

#6: 28 Sep 2020

Attendees

Key Decisions

• Scheduler HA
  • Locking discussion: Document how Scheduler HA would work with a HA database configuration, such as an Active / Passive database configuration with multiple read replicas, but a single writer.  
  • Add MySQL 8 to the our CI pipeline https://github.com/apache/airflow/issues/11164
  • Progress update: Ash said that the unit tests should be green by tomorrow. One thing left is the change_state_for_to_without_dagrun - currently called for every time (expensive), will change to call every 30 seconds. PR is feature complete after that is done and will no longer be "in draft". Will need to move first functionally complete build to next week instead of current week, to give time for all unit tests to pass and for reviews. 
  • Benchmarks: Will be run based on this branch, since the current benchmarks are based on an initial draft branch before rebase with master. 

• API
  • Task Instance Endpoint: Only thing open. Kaxil to speak with Kamil to get status. 
  • Existing permissions map to UI: WIP PRs. Feedback requested by James on this PR https://github.com/apache/airflow/pull/11158/
  • Migrations: Is there a different migration process needed? Can be run as a standard alembic migration. 
  • Clients: QP said that the GO client based on the REST API is already completed and he is using it as part of his airflow-terraform module.
• UI Improvements
  • New HomePage: Ryan has updated the UX (look) and shared on slack. Split Actions and Buttons, using Google Material Fonts. Feedback from meeting was very positive. PR to be final tomorrow. 
• Splitting Providers Package
  • Separate provider packages from core: Need some help to get this wrapped up. https://github.com/apache/airflow/issues/11163
  • Build optimization: This would really help speed-up builds by only running tests for changed providers https://github.com/apache/airflow/issues/10507
  • SemVer: Kaxil to send email to dev list confirming the decision in the meeting about using SemVer (lazy-consensus)
• Backport Providers in next 1.10.x release
  • EmailOperator: Location has changed (moved to core) 
  • CNCF.Kubernetes providers: Need to be checked if these can be released. Insert github issue link created by Kamil for core operators
• Timeline update:
  • Functionally complete build: Will be on the week of Oct 5th, instead of this week as described above in the Scheduler HA notes. 

Things to discuss next

#7: 5 Oct 2020

Attendees

Key Decisions

• Scheduler HA

  • Progress

    • MySQL 8 tests have been added
    • Adoption of orphaned tasks have been added to Kubernetes and to the new CeleryKubernetesExecutor (PR)
    • All (but one) Tests are passing
    • Instructions to run the PR with Multiple Schedulers using docker-compose have been added (link to the comment)
    • Received some reviews on Monday -- Should be addressed by today/tomorrow
  • All Big/Critical Concerns have been addressed (escape hatch for locks, MySQL 8 tests, Moving callbacks to DagFileProcessor)
  • Room to improve code readability which isn't a blocker and would be addressed in follow-up PR
  • Ash is in talks with Kevin to test his Giant DAG to verify Scheduler Performance to address one of Kevin Yang's concern
  • There would be a separate PR for retrying transactions to avoid deadlocks as suggested by MySQL guide on handling deadlocks
• API
  • Kamil and Omair are hoping to complete work on the Task Instance Endpoint PR by the end of this week
  • James is working to map existing permissions to the UI, hoping to complete by end of the week too. Link to PRs
• Backport Providers
  • Backport packages 2020.10.5 will be released without cncf.kubernetes. We will create a separate release for cncf.kubernetes that will address the issues Kamil found in RCs. Link to VOTE thread.
• Airflow 1.10.13 Release
  • Will be released as per the planned timeline (Airflow 2.0 - Planning) i.e. Last Week of October. 
• Airflow 2.0 Alpha
  • Revisit on Thursday based on the progress with Scheduler HA & API whether we would be able to release first alpha this Friday.
•  Others:
  • Kaxil to create Github Issues for the feedback received on the Documentation Improvement Thread and create threads asking for VOTES for previously discussed items like removing pickles.
  • Vikram will write up, share and all can review on what we mean by ALPHA and BETAs.
  Doc improvements