Page History

#1: 10 Aug 2020 - Planning & Scoping

...

• Scheduler HA

  • Progress

    • MySQL 8 tests have been added
    • Adoption of orphaned tasks have been added to Kubernetes and to the new CeleryKubernetesExecutor (PR)
    • All (but one) Tests are passing
    • Instructions to run the PR with Multiple Schedulers using docker-compose have been added (link to the comment)
    • Received some reviews on Monday -- Should be addressed by today/tomorrow
  • All Big/Critical Concerns have been addressed (escape hatch for locks, MySQL 8 tests, Moving callbacks to DagFileProcessor)
  • Room to improve code readability which isn't a blocker and would be addressed in follow-up PR
  • Ash is in talks with Kevin to test his Giant DAG to verify Scheduler Performance to address one of Kevin Yang's concern
  • There would be a separate PR for retrying transactions to avoid deadlocks as suggested by MySQL guide on handling deadlocks
• API
  • Kamil and Omair are hoping to complete work on the Task Instance Endpoint PR by the end of this week
  • James is working to map existing permissions to the UI, hoping to complete by end of the week too. Link to PRs
• Backport Providers
  • Backport packages 2020.10.5 will be released without cncf.kubernetes. We will create a separate release for cncf.kubernetes that will address the issues Kamil found in RCs. Link to VOTE thread.
• Airflow 1.10.13 Release
  • Will be released as per the planned timeline (Airflow 2.0 - Planning [Archived]) i.e. Last Week of October. 
• Airflow 2.0 Alpha
  • Revisit on Thursday based on the progress with Scheduler HA & API whether we would be able to release first alpha this Friday.
•  Others:
  • Kaxil to create Github Issues for the feedback received on the Documentation Improvement Thread and create threads asking for VOTES for previously discussed items like removing pickles.
  • Vikram will write up, share and all can review on what we mean by ALPHA and BETAs.

...

• 1.10.13 to be released this week
• First version of upgrade-check to be released this week
• Composer team + Polidea will test 2.0.0beta3 next week on MySQL 5.7 + MySQL 8
• Don’t need new beta for testing providers, Committers / PMC will test it themselves (Providers’s Packages discovery + Connections / Extra Links)
• Jarek is working on "Registering Extra link & Connections for Providers"

#15: 30 Nov 2020

Attendees

Discussion and Key Decisions

• Bridge release  
  • Airflow 1.10.14 to be built this week and vote started to release next week.
  • Airflow 1.10.13 was released last week, but critical bug discovered after release.
• Upgrade Checks
  • First version of upgrade-check released last week.
  • Additional checks and enhancements to support custom checks being added.
• Airflow DB upgrade
  • The topic of Airflow Database migration was brought up and discussed.
  • We agreed that it would be ideal to recommend this approach for upgrading from 1.10.14 to 2.0, but we need to add a FAQ around how long this database upgrade would take and what possible steps could be taken to reduce this time.
  • For example, by running a script or a DAG to reduce the amount of task history before the database upgrade.
  • A collection of FAQs around this would be important to have as part of the 2.0 release. 
  • Vikram to add issues to capture this.
• Airflow 2.0 Release Candidate: 
  • Overall feedback on Airflow 2.0 core is positive, but polishing still needed.
  • Targeting RC1 build creation to start in the middle of next week.
  • Command line PR not yet merged, but would be good to have in RC1. 
  • Documentation work is ongoing, would be good to clean up before RC1 release.
• Milestone Issue Cleanup
  • Issues currently tagged for 2.0 RC1 should be cleaned up.
  • Those which are not core such as Providers or upgrade scripts, should be tagged for those milestones rather than the 2.0 core RC1 milestone. 
  • All of us to do this cleanup this week and review any open items next Monday. 

#16: 7 Dec 2020

Attendees

Discussion and Key Decisions

• Airflow 2.0 Release Candidate: 
  • Targeting 2.0.0rc1 Wednesday - 9 Dec.
• Upgrade Checks
  • Targeting 1.1.0rc1 Friday - 11 Dec.
• Providers 1.0.0rc1
  • Targeting with 2.0.0rc1 on Wednesday - 9 Dec.
• Airflow 1.10.14

  • 1.10.14rc4 has been cut and submitted for a vote. If vote passes, 1.10.14 would be released on Thursday 10 Dec

• Deprecating Features
  • Follow strict SEMVER.
  • Features won’t be removed until a major version even if they are deprecated
  • Document a policy about that in CONTRIBUTING.rst or somewhere more appropriate
  • Ash to write a proposal about having a major release (e.g 3.0 release) with just removed features

#17: 14 Dec 2020

Attendees

Discussion and Key Decisions

• Airflow 2.0
  • If the vote succeeds, Airflow 2.0 will be released on Thursday, 17 December
• Provider 1.0.0 
  • Apache Airflow Providers 1.0.0 has been released: https://pypi.org/search/?q=%22apache-airflow-providers%22&o=
• Airflow upgrade-check
  • Airflow upgrade-check 1.1.0 has been released: https://pypi.org/project/apache-airflow-upgrade-check/1.1.0/
• Snowflake Issues
  • Snowflake monkey-patches urllib -- which causes issues for example when using S3 logging in 2.0: "'SSLSocket' has no attribute Connection"
  • GH links here: 
  • https://github.com/snowflakedb/snowflake-connector-python/issues/324
  • https://github.com/apache/airflow/issues/12881
  • Escalate and Push Snowflake to fix it and release a new version as early as they can
• Dev calls after 2.0
  • Fortnightly meeting (once every two weeks) each Wednesday at 7:30 PM GMT after 2.0 is released, starting from 6 Jan 2021.
  • Check https://www.timeanddate.com/worldclock/fixedtime.html?msg=8&iso=20210106T1930 to find the time in your Timezone