Proposed Agenda
- Catch-up on action items from last call
- Development Updates and Presentations (Please keep the updates to less than 10 minutes):
- Discussion topics:
Attendees
Vikram Koka |
Shahar Epstein |
Elad Kalif |
Kaxil Naik |
Jarek Potiuk |
Aritra Basu |
Pavan Kumar |
Amogh Rajesh Desai |
Jens Scheffler |
Buğra Öztürk |
Rajesh Bishundeo |
Niko |
Karthikeyan Singaravelan |
Ash Berlin-Taylor |
Zach Gottesman |
Daniel Standish |
Shubham Raj |
Vishnu Chilukoori |
Dennis Ferruzzi |
Summary
- Development Updates:
- Update on the 3.0.1 release Unknown User (kaxilnaik)
- Kaxil mentioned that over 200 commits are part of 3.0.1 RC1 including many usability improvements.
- He said that the voting for RC1 ends on Monday, the 12th of May.
- Jarek added that we would probably need an RC2 because of the inclusion of an older version of an Amazon provider.
- 2.11 release scope / timing Unknown User (kaxilnaik)
- Kaxil mentioned that the decision between 2.11 vs. 2.10.x was made because of several features and changes in the 2.x branch, which meant beyond a patch.
- Specifically, he mentioned the new Timetable addition as well as the PR for making sure that the metrics are aligned for consistent reporting.
- Another reason for the feature release was also to add support for "airflow config lint" which enables validation of the airflow configuration before upgrading to Airflow 3.
- Discussion Topics:
- 3.0 release feedback / discussion topics (All)
- Vikram shared a quick update on the Airflow downloads by version over the last two weeks since Airflow 3 was released.
- Vikram highlighted that Airflow 3 has seen significant adoption with about 7.5% of the total Airflow downloads. The highest downloaded version continues to be 2.10, which is the last release on the 2.x release stream, with about a third of all downloads. Of course, these metrics should be taken with the usual grain of salt, because they are susceptible to image caching and CI processes.
- Elad asked where features "missing" in Airflow 3 were documented. Vikram said that these were documented as breaking changes within the Upgrading to Airflow 3 doc. The team identified a couple of missing elements, which Elad volunteered to add.
- Jarek said that there was significant user confusion around upgrading to 3.0 with respect to DAG writing practices, specifically around imports. After much discussion, the conclusion was that one of the core problems here was the UX around doc versions on the website, being too subtle and not obvious to users about the differences between 2.x and 3.0 syntax. The action item for Jarek and Elad was to look at the docs UX to see if this could be made more obvious.
- 3.1 release scope / timing discussion Unknown User (vikramkoka)
- In the follow-up discussion about the scope of 3.1, Vikram explained that the Airflow 3.1 scope was largely based on user feedback of 3.0 including the prior discussion about missing features.
- The early draft of Airflow 3.1 scope is documented on the wiki for discussion.
- Vikram mentioned that Dennis couldn't make the call today, but asked for reviews on his PR which was early work towards Deadline Alerts, which replaces SLA alerts in Airflow 2.x and is currently scoped for Airflow 3.1. Rajesh added that he was confident with the timing, since Dennis and others were actively working on this and would be ready to update the team in two weeks' time.
- Jarek raised the question a
- Other language support
- Ash shared an update on the Go SDK. He shared a link to the branch with the draft incarnation and said that a PR would be coming soon.
- Vikram shared an update with the early Java SDK branch. Jens chimed in referencing the offline discussion with Vikram around leveraging the Edge API for these other languages.
- The action item was for Jens, Ash, Kaxil, and Vikram to discuss async on any changes needed to the Edge provider.
- Ash raised a concern that we may not see significant adoption until the DAGs were in other languages as well. Elad and Jarek argued against the viewpoint feeling that a single task in a different language would be very powerful for application integration. Elad said that "The notion of executing a task directly without a wrapper DAG is a request I hear from time to time (already)".
- FAB upgrades (4.6.2 first, 5.0.0 later) Unknown User (potiuk)
- Jarek raised the FAB upgrade issue which we had deferred a couple of months ago to post 3.0. He said that we are in a much better situation now, but that he needed help with the SqlAlchemy migrations and would follow up on the dev list.
- As part of this conversation, there was a discussion about dropping support for Python 3.9 early. However, there was no appetite for that from the attendees because of all the complexity around doing this selectively for Core Airflow vs. Providers. This was going to be conveyed back to the dev list discussion by Jens.
- Example DAGS Unknown User (jscheffl) Unknown User (potiuk)
- Jens and Jarek brought up the issue with the Example DAGs which have grown to the point of having become too confusing, because they are used for two different purposes: One for developers testing Airflow before release, and second for users to learn how to use Airflow.
- There was clear agreement on separating the examples for those two use cases and Jarek was going to take the lead on that.
- Jens was going to take the lead on making the true end user example DAGs more useful with a story telling concept for a broad use case.
- Continuing dev calls
- Vikram suggested that we make these dev calls every two weeks going forward, rather than weekly.
- There was immediate agreement on this.
- Action items:
- Airflow docs UX update to make versions more obvious (Elad, Jarek)
- Update on Deadline Alerts AIP progress and Github board (Dennis, Rajesh)
- Edge executor for other languages (Jens, Ash, Kaxil, Vikram)
- Update devlist about no-go on Python 3.9 early drop support (Jens)
- Get help on SqlAlchemy upgrade changes (Jarek)
- Example Dags update (Jens, Jarek)
- Make dev calls to be every two weeks (Vikram)