Airflow Dev Calls

Dev Calls Calendar

We have regular calls between Airflow Developers (PMC Members, Committers and community members) to discuss various issues – progress on next releases, issue triage, tracking milestone items, etc.

6 Jan 2021

Attendees

Summary

• Airflow 2.0.1 / Open Issue from 2.0:
  • Milestone: Airflow 2.0.1 Milestone · GitHub
  • Add ‘priority’ labels to all the open issues in 2.0.1 Milestone
  • Release 2.0.1 as soon as the reported bugs are fixed
  • Positive feedback from Scribd – Going PROD with 2.0 in 2 weeks. No major bugs reported by Astronomer, Google / Polidea and Amazon.
• Upgrade Checks
  • Release 1.2.0 soon with a fix for https://github.com/apache/airflow/issues/13293
  • Add a Rule to Check version of Database (MySQL / Postgres / SQLite)
• Providers
  • Elasticsearch and Google (breaking change - 2.0.0) providers would be released soon or at least RCs would be cut next week
• Release Process - Semantic Versioning
  • Changing the version of dependency should not count as breaking change.
  • Ash is working on the doc which will be PRd to Airflow soon
  • Still no unanimous agreement on whether "Major Version should contain new features or just removal of deprecation"
  • Highlight this process to all the committers (chances are some committers are not regularly active). The most important one is that we will strictly follow SEMVER.
• Airflow 2.1 - cadence, candidates and planning
  • Patch releases 2-3 weeks (Might be driven by the bugs and their severity)
  • Minor releases would be released every two months
  • No cadence for Major releases!
  • What happens to the PR that introduces breaking changes??
  • We should not merge that PR until it is a breaking change, instead we should make the PR backwards compatible
  • Or wait until next major release (assign appropriate Github Milestone)

20 Jan 2021

Attendees

Summary

• Airflow 2.0.1:
  • Milestone: Airflow 2.0.1 Milestone · GitHub
  • Target week of Feb 8 for release
• Providers Release
  • Next set of Providers will be released next week once https://github.com/apache/airflow/pull/13767 is merged with appropriate SEMVER version (e.g Google would be a major release while ES would be a patch release)
  • Backport Providers will be released for the ones that have some significant changes after or at the same time as the providers
  • Snowflake released a new version (v2.3.8) and the Snowflake provider which will use this version will be released too as currently all versions for Snowflake providers are YANKED.
    • This will be released after https://github.com/apache/airflow/pull/12188 is merged as it contains some incompatibilities with Azure Blob Storage
• Upgrade Checks
  • Release 1.2.0 soon with a fix for https://github.com/apache/airflow/issues/13293
  • Add a check for importing Operator/Sensor via Plugins if it does not already exist
  • Add docs in 1.10.15 about using upgrade-checks to upgrade to 2
• Helm Chart Release
  • We need to agree on what we call the first version: 0.0.1 vs 2.0.0
  • We need to review the current Helm chart before it is released as a lot of changes have been merged since it was first merged.
  • Current Test Coverage is not that great - ideally, we would like to cover more test cases before it can be released 
  • We hope to release it after 2.0.1 so bugs fixes are in by then and 2.0.1 can become the default version for the Helm Chart

3 Feb 2021

Attendees

Summary

• Airflow 2.0.1:
  • Milestone: Airflow 2.0.1 Milestone · GitHub
  • Will be released on Feb 8
• Providers Release
  • Providers will be released on Feb 8
  • Backport Providers should also be released as along with Providers, assuming vote passes
• Upgrade Checks
  • 1.2.0 will be released soon after Airflow 2.0.1 once the vote passes
• Apache Airflow 1.10.15
  • Milestone: https://github.com/apache/airflow/milestone/22?closed=1
  • CI is fixed - v1-10-stable passing
  • A release date has not been decided yet
• Improve Airflow Observability
  
  • John to create Github Issue around what metrics would be useful to avoid failures early from an Operational point of view. Based on that we can discuss and recommend using existing metrics or add missing metrics.
• Make Doc contributions easier
  • We should have a separate section in CONTRIBUTING.rst for users who want to create PR for only doc-changes.
  • Currently, it is difficult for new contributors to understand how to add cross-links, using sphinx, building docs locally, debugging spelling-check errors.
  • Adding some recipes/templates in the CONTRIBUTING.rst with code to create Cross-references, adding links, spell-check, etc using Sphinx would make it easy.

17 Feb 2021

Attendees

Summary

• Pending Item:
  
  • John Jackson to create Github Issue to improve Airflow's observability story
  • Vikram Koka to create Github Issue/PR to add a guide to help new contributors make doc-only contributions
  • Ash Berlin-Taylor to complete the Draft PR on Release Docs / Version Policy. Jarek Potiuk to follow it up to add version policy for Providers
• Github Issues Stats
  • Vikram presented some stats and charts around our Github Issues
    • Kubernetes was an area that needs attention. 
• Apache Airflow 1.10.15
  • Milestone: https://github.com/apache/airflow/milestone/22?closed=1
  • Tentative Date: 1 Mar 2021
• Helm Chart
  
  • Project Board: https://github.com/apache/airflow/projects/7 to track the work
  • Tentative Release Date: Early April
  • The first release will be 1.0.0

Agenda for next week:

• TaskGroups for Tree View (Kevin Yang)
• Provider Release Cadence
  • Ad-hoc Release vs Regular Releases