The dev calls are open for anyone in the community to join and help in the Airflow 3 efforts. 

Schedule: Fortnightly (once in 2 weeks) on Thursday at 4pm GMT/UTC

Table of Contents

• 8 May 2025
  • Proposed Agenda
  • Attendees
  • Summary
• 22 May 2025
  • Proposed Agenda
  • Attendees
  • Summary
• 5 June 2025
  • Proposed Agenda
  • Attendees
  • Summary
• 26 June 2025
  • Proposed Agenda
  • Attendees
  • Summary
• 3 July 2025
  • Proposed Agenda
  • Attendees
  • Summary
• 17 July 2025
  • Proposed Agenda
  • Attendees
  • Summary
• 31 July 2025
  • Proposed Agenda

8 May 2025

Proposed Agenda

Attendees

Summary

• Development Updates:
  • Update on the 3.0.1 release Kaxil Naik 
    • Kaxil mentioned that over 200 commits are part of 3.0.1 RC1 including many usability improvements.
    • He said that the voting for RC1 ends on Monday, the 12th of May. 
    • Jarek added that we would probably need an RC2 because of the inclusion of an older version of an Amazon provider. 
  • 2.11 release scope / timing Kaxil Naik 
    • Kaxil mentioned that the decision between 2.11 vs. 2.10.x was made because of several features and changes in the 2.x branch, which meant beyond a patch. 
    • Specifically, he mentioned the new Timetable addition as well as the PR for making sure that the metrics are aligned for consistent reporting. 
    • Another reason for the feature release was also to add support for "airflow config lint" which enables validation of the airflow configuration before upgrading to Airflow 3. 
• Discussion Topics:
  • 3.0 release feedback / discussion topics (All)
    • Vikram shared a quick update on the Airflow downloads by version over the last two weeks since Airflow 3 was released. 
    • Vikram highlighted that Airflow 3 has seen significant adoption with about 7.5% of the total Airflow downloads. The highest downloaded version continues to be 2.10, which is the last release on the 2.x release stream, with about a third of all downloads. Of course, these metrics should be taken with the usual grain of salt, because they are susceptible to image caching and CI processes.
    • Elad asked where features "missing" in Airflow 3 were documented. Vikram said that these were documented as breaking changes within the Upgrading to Airflow 3 doc. The team identified a couple of missing elements, which Elad volunteered to add. 
    • Jarek said that there was significant user confusion around upgrading to 3.0 with respect to DAG writing practices, specifically around imports. After much discussion, the conclusion was that one of the core problems here was the UX around doc versions on the website, being too subtle and not obvious to users about the differences between 2.x and 3.0 syntax. The action item for Jarek and Elad was to look at the docs UX to see if this could be made more obvious.
  • 3.1 release scope / timing discussion Vikram Koka  
    • In the follow-up discussion about the scope of 3.1, Vikram explained that the Airflow 3.1 scope was largely based on user feedback of 3.0 including the prior discussion about missing features.
    • The early draft of Airflow 3.1 scope is documented on the wiki for discussion. 
    • Vikram mentioned that Dennis couldn't make the call today, but asked for reviews on his PR which was early work towards Deadline Alerts, which replaces SLA alerts in Airflow 2.x and is currently scoped for Airflow 3.1. Rajesh added that he was confident with the timing, since Dennis and others were actively working on this and would be ready to update the team in two weeks' time.  
    • Jarek raised the question a
    • Other language support
    • Ash shared an update on the Go SDK. He shared a link to the branch with the draft incarnation and said that a PR would be coming soon. 
    • Vikram shared an update with the early Java SDK branch. Jens chimed in referencing the offline discussion with Vikram around leveraging the Edge API for these other languages.
    • The action item was for Jens, Ash, Kaxil, and Vikram to discuss async on any changes needed to the Edge provider. 
    • Ash raised a concern that we may not see significant adoption until the DAGs were in other languages as well. Elad and Jarek argued against the viewpoint feeling that a single task in a different language would be very powerful for application integration. Elad said that "The notion of executing a task directly without a wrapper DAG is a request I hear from time to time (already)". 
  • FAB upgrades (4.6.2 first, 5.0.0 later) Jarek Potiuk 
    • Jarek raised the FAB upgrade issue which we had deferred a couple of months ago to post 3.0. He said that we are in a much better situation now, but that he needed help with the SqlAlchemy migrations and would follow up on the dev list. 
    • As part of this conversation, there was a discussion about dropping support for Python 3.9 early. However, there was no appetite for that from the attendees because of all the complexity around doing this selectively for Core Airflow vs. Providers. This was going to be conveyed back to the dev list discussion by Jens.
  • Example DAGS Jens Scheffler  Jarek Potiuk 
    • Jens and Jarek brought up the issue with the Example DAGs which have grown to the point of having become too confusing, because they are used for two different purposes: One for developers testing Airflow before release, and second for users to learn how to use Airflow. 
    • There was clear agreement on separating the examples for those two use cases and Jarek was going to take the lead on that. 
    • Jens was going to take the lead on making the true end user example DAGs more useful with a story telling concept for a broad use case.  
      • UPDATE:
        • Planning Page: Examples Refurbish
        • Discussion in Slack: #sig-examples 
  • Continuing dev calls
    • Vikram suggested that we make these dev calls every two weeks going forward, rather than weekly.
    • There was immediate agreement on this. 
• Action items:
  • Airflow docs UX update to make versions more obvious (Elad, Jarek)
  • Update on Deadline Alerts AIP progress and Github board (Dennis, Rajesh)
  • Edge executor for other languages (Jens, Ash, Kaxil, Vikram)
  • Update devlist about no-go on Python 3.9 early drop support (Jens)
  • Get help on SqlAlchemy upgrade changes (Jarek)
  • Example Dags update (Jens, Jarek)
  • Make dev calls to be every two weeks (Vikram)

22 May 2025

Proposed Agenda

Attendees

Summary

• Catch-up on action items from last call:
  • Docs site update (Elad, Jarek)
    • Elad said that they could not make much progress on the Docs site update over the last couple of weeks, but would have an update by the next dev call. 
    • Jarek added that there has been significant improvements to the Doc publishing process, led by Pavan.
    • The team agreed with that and gave a shoutout to Pavan, Jarek, and everyone who had contributed to these improvements. 
  • Example Dags update (Jens, Jarek)
    • Jens talked about the small team who have gathered together to start working on improvements to the Example DAGs.
    • Jens shared the wiki page which was being used to track these ideas. 
    • There was a discussion around whether "testing DAGs" are useful to end users and if they should be tagged as "dual use" to reflect that they can be used as an example.
• Development Updates:
  • Deadline alerts AIP-85 (Dennis Ferruzzi, Rajesh B)
    • Dennis and Ramit gave an update on Deadline alerts which included a long, in-depth technical discussion around the implementation options. 
    • There was significant discussion around where the loop to check for alerts should be run as well as where the dag-level callbacks should be run. 
    • The view from the team was that system monitoring i.e. if the scheduler is down is out of scope for this effort and should be done outside of Airflow, i.e. if Schedulers (or Workers) are down, no SLAs would be fired. 
    • After the extended discussion, the summary was that we aligned on a variation of option 1 from the email, with the following elements:
      • Scheduler loop would be used for checking SLA misses
      • If an async notifier is specified, the callback would be run in the triggerer
      • If a sync notifier is specified, the callback would be run on a worker
      • We recommend async callbacks using the builtin Airflow async notifiers
  • Update on the 3.0.x and 2.11 releases Kaxil Naik 
    • Kaxil mentioned that we released 3.0.1 and 2.11 already and that we would be cutting 3.0.2 in about two weeks.
• Discussion Topics:
  • UI Internationalization
    • Brent shared that we had just merged a PR to support internationalization in the UI.
    • There was a lot of excitement around this. 
    • There was an early discussion on what terms should NOT be translated such as "DAG" or "Xcom".
    • This was to be followed up offline.
  • Golang SDK
    • Ash shared that he had raised a PR for an early incarnation of the Golang Task SDK and requested feedback. 
  • Airflow 3 feedback:
    • Pavan shared that he was at the Astronomer roadshow in London the day before and the feedback from people was very positive. 
    • Quote: "People are amazed with Airflow 3 features, dag versioning, Event driven scheduling. everyone looking to upgrade AF3, that's what i hear at least from the people i met :)"

5 June 2025

Proposed Agenda

Attendees

Summary

• Catch-up on action items from last call:
  • Docs site update (Jarek)
    • Jarek shared a quick, AI generated view of what an updated version selection UI could look like on the docs site. 
    • Jarek also articulated an idea of showing a banner (either left or top) to highlight which Airflow version was being viewed at the moment. 
    • Vikram brought up a point of possibly only showing the major.minor versions of docs, not showing the full list of patch versions as well, to avoid an overwhelming list of choices.
    • Vikram also brought up the issue of the Airflow Core version vs. Task SDK Version. Ash said that most of the docs would probably end up associated with the TaskSDK version, but we weren't there yet. 
    • Jens highlighted that we need to be careful to not overwhelm an initial user with the internal Airflow complexity. 
    • Ended the discussion with a desire to get feedback from UX perspectives, before proceeding forward.
• Development Updates and Presentations
  • UI plugins - Pierre
    • Pierre gave a quick update regarding the status of the UI plugins AIP. 
    • The first kind of plugins being worked on are the iframe plugins and the front-end will take care of rendering them. Templates can be used here with attributes including elements as TaskID and DagID, so that the display can show the context. 
    • The second kind of plugins will be to support React plugins. Additionally, Brent is thinking about dev tools to facilitate the creation of these plugins. 
    • All of this is aimed for 3.1 and quite confident on timing. 
    • Jens raised the question of whether these should initially be marked as "experimental", so that we have the flexibility to change the interface based on initial feedback. 
  • Deadline Alerts - Dennis
    • Dennis raised a database design question for the team encountered during the development of the AIP.
    • Specifically whether to only keep the list of deadline alerts to be checked in the database table, or to keep historical records (such as deadlines met or not met) as well. The point was regarding the performance impact of keeping more data vs. the benefits of keeping this information. 
    • Pierre raised the point that keeping more than "currently in flight" deadlines would be useful from a UI/UX perspective.
    • Dennis was wondering if an index would be sufficient to mitigate performance. Ramit suggested using a view and Vikram suggested that in the worst case, you could have a separate table.  
    • After some discussion, the action item was for Dennis to make the decision using tests / benchmarks based on expected sizes. 
  • Internationalization - Shahar
    • Shahar presented the proposed internationalization policy to address challenges with maintaining translations across 300 languages with only 70 committers. 
    • Shahar mentioned that German, Polish, Dutch, and Mandarin translations have already been merged.
    • And that, Hebrew and Arabic translations are pending review.
    • The presentation covered details such as assigning two roles for each locale: a translation owner responsible for language aspects and a code owner handling technical aspects, with translation sponsors to potentially be non-committers. 
    • It also referenced a process for release managers to notify code owners about missing translation items as part of cutting the release. 
    • The release process impact and timing prompted significant discussion with the team expressing concerns about increasing the burden on release managers. 
    • Jarek suggested starting the release process a week earlier to help accomplish this.
    • Shahar explained that the fallback policy was to use English when terms are not translated, but this raised UX concerns. 
    • The team agreed to limit the number of languages to start with to reduce overhead and implement a system for periodic reminders about missing translations. 
    • Shahar took the action item to follow up with potential refinements to reduce release friction.

26 June 2025

Proposed Agenda

Attendees

Summary

• Catch-up on action items from last call:
  • Docs site update (Jarek)
    • Jarek mentioned that there is an issue created for the work here.
    • The docs build process is a lot better now because of the work led by Pavan, but the actual result still needs work.
    • Jarek also spoke with Kamil about the original work done on the docs site and his availability to help. 
  • Update regarding internationalization impact on release process (Shahar)
    • Shahar said there isn't any change to the process from prior discussions. 
    • The expectation is for the Release Manager to announce that we will cut a new release and that serves as a notification for the code owner to finish translation by a particular date. 
    • The medium term intent is to publish translations outside of "Core Airflow". The initial concept here is to have translations released similar to provider releases and not to be tied to the core Airflow software release. This could be just a file release or even a data update to the database, to be figured out. 
    • There was a lot of enthusiasm from the team about how this project has happened so quickly and all the community involvement here. 
• Development Updates and Presentations (Please keep the updates to less than 10 minutes):
  • UI Plugins Update (Brent, Pierre)
    • Brent and Pierre did a quick demo of the UI plugins work and shared the progress to date. 
    • This included the common abstraction for the UI plugin regardless if the UI should be in a new tab or an iFrame rendering
    • The team was enthusiastic about the great progress here.
  • Deadline Alerts: Update on conclusion for database tradeoff (Dennis)
    • Ramit shared that the development completion target for Deadline alerts is the end of July
    • Ramit also shared that right now when a DAG is created with a deadline alert, the deadline would be created and stored in the database
    • The scheduler loop change work is upcoming and the database change approach will be figured out as part of that. 
    • Brent asked about needing to sync on the API endpoints for the same. Dennis to sync with Brent over the course of the next week.
• Discussion topics:
  • Multi-team discussion from dev list - (Jarek)
    • There was long spirited discussion around the multi-team effort. Jarek opened by asking for comments. 
    • Vikram went first, saying that he wanted to make sure that we were not trading off the simplicity of Airflow starter experience for individual data engineers with the complexity needed for large organizations wanting granular operational procedures. Vikram asserted that as an open source project, viral adoption has been the key to Airflow's success and that simplicity was important to maintain and in fact make better. 
    • Jarek immediately agreed with that assertion and said that was a key point in his desire to also simplify this AIP to multi-team, from the original, far more complex, multi-tenancy proposal. 
    • Vikram also said that he was surprised by the lack of reference to AIP-82 Event driven scheduling, since this was such a commonly used mechanism for segmenting Airflow deployments across teams from a ownership perspective, following classic distributed system principles. 
    • Jens also chimed in saying that we should perhaps articulate this better in the context of how to set up Airflow for large enterprises. 
    • There was a significant level of debate around the complexity vs. simplicity tradeoff, with Rajesh stating this is probably the most common requirement that MWAA hears from their customers. Niko also chimed in saying that we should be willing to accept some tradeoffs to simplicity. 
    • There was a further discussion on what needs in priority order needed to addressed here, with Kaxil raising the point that this raised significant operational considerations and he wanted to make sure that we were not getting in over our skis in complexity with quota management, etc. 
    • This ran out of time without an explicit decision in the meeting itself.
    • Jarek concluded by asking people to comment back in the AIP soon, so that feedback could be addressed in writing. 
  • Call for actions ...ah.. review! Example Dags! (Jens)
    • Jens raised a "call for action", asking people to help review the work on the Example DAGs and to contribute

3 July 2025

Proposed Agenda

Attendees

Summary

• Catch-up on action items from last call:
  • None
• Development updates and presentations: 
  • Airflow 3 adoption (Vikram)
    • Vikram shared that the download numbers show encouraging signs of 3.0 adoption, with about 12% of all downloads for the last month being for 3.0 placing it in third place, with 2.10 in the lead with 29% and 2.6 (still puzzling) in second place with 16%.  
    • Vikram also noted that 2.11 which was released after 3.0 is at 5% of downloads, which showed that more people were willing to go to 3.0 directly from earlier 2.x versions. 
    • Jens noted that it was still a very good thing that we had released 2.11, since 5% of people had upgraded to it from earlier 2.x versions, presumably because they were not yet ready to upgrade to 3.0. 
    • The team noted that for several organizations there were a couple of functional gaps, notably UI Plugins and "run as user" which blocked the upgrade to 3.0. Amogh responded that "run as user" is available as part of Airflow 3.0.3.
    • Vikram noted that the two big known gaps i.e. UI plugins and Deadline Alerts are targeted for the 3.1 release.
    • The team noted that we should make an effort to clean up any such small blockers in the 3.1 release. 
    • The ask for the community is to please highlight such gaps encountered in the dev list or the Airflow 3 dev slack channel. 
  • CLI update (Bugra)
    • Bugra walked through the Github project for this AIP as well as the current status of the work. 
    • Bugra stated that the project would be ready to be integrated into the release process in a couple of weeks.
    • Bugra also volunteered to help in the release process as needed. Jarek pointed out that some additional work may be needed around tests as part of getting ready for release. 
  • Airflow 3.1 update (Vikram)
    • Vikram shared the overall scope and timing of the Airflow 3.1 release as documented on the wiki page. 
    • The one area of significant discussion was the Human in the loop as detailed below.
• Discussion topics:
  • Human in the loop
    • Jens started the discussion here by pushing for the Human in the loop feature to be built and released as a separate provider, not within the standard provider package.
    • Kaxil vocally advocated for this to be in Core Airflow for simplicity and walked through the initial PR created by Wei to illustrate his rationale. 
    • Jarek vehemently agreed with Kaxil. 
    • Pierre also agreed with Kaxil, saying that he would prefer UI plugins to be only used by users, not by us as Airflow contributors. 
    • This then led to the discussion that we have gone too far with the Provider separation from Core Airflow. Jed brought up the point that he would like to get the Executors out of the Providers into a different package. 
    • The discussion ended with the decision to make the "core" Human in the loop functionality as part of Core Airflow. 
  • Code sharing discussion (Ash)
    • The prior discussion also led to the code sharing discussion between Task SDK and Core Airflow, following the thread brought up by Ash in the dev list. 
    • This was a great discussion with a lot of participation from everyone on the call. 
    • Ash to summarize this.

17 July 2025

Proposed Agenda

Attendees

Summary

• Catch-up on action items from last call:
  • None
• Development Updates
  • UI Plugins Update (Pierre and Brent)
    • Pierre shared that the React plugins implementation is nearly complete, including embedded web applications now working inside the Airflow UI. 
    • Pierre also said that key technical challenges have been resolved including react library sharing between plugins and Airflow Core.
    • Pierre raised one issue which had been open about where to keep the CLI/Python scripts to bootstrap React project. Based on team feedback, it was resolved to have those live within dev-tools. 
    • Brent gave a quick demo of the React plugins working in the Airflow UI
    • Outstanding work on this stream before the 3.1 release includes documentation and examples to demonstrate both iFrame and React app integration, as well as CSS fine-tuning for wider screen layouts. 
  • Task SDK Update (Kaxil, Ash, and Amogh)
    • Kaxil shared that multiple parallel work streams are active right now for this AIP
    • Ash is prototyping the common object sharing approaches (symlink vs. rendering). 
    • Utility code migration is happening based on that to appropriate locations (task SDK vs. shared). Serialization structure changes are also being done to support this. 
    • The Core Operator cleanup is happening, largely based on TP's work. Base Operator is now removed from Airflow models. Similar removal planned for Map operator and other core Operators. 
    • Branch Operator bug identified and is being addressed. 
  • Python 3.13 (Jarek)
    • Jarek shared that Python 3.13 is now available for all users and gave a quick demo of the same. 
    • Jarek also said that the Flask Application Builder (FAB) dependency has now been moved and a truly FAB-less Python 3.13 deployment is possible. 
    • Jarek also shared that Python 3.13 fixed a long-standing file description duplication bug. This bug could explain the recent "too many files open" issues in supervisor tasks. 
  • Deadline Alerts Update (Ramit and Rajesh)
    • Ramit shared that he had done performance testing on different approaches using a standalone Python script on Postgres, with 10 million rows. 
    • He tested 3 approaches: Two separate tables, view-based single table, and indexed single table. 
    • The index approach (state + deadline time) had the best performance and therefore led to the implementation approach of adding a state column to the deadlines tables. 
    • Ramit also shared that async callbacks would be run on the triggerer and synchronous callbacks on the workers. 
    • Rajesh shared that the deadline detection code is nearly complete, with the Slack and SMTP notifiers being implemented as async functions. Documentation is also in progress, with an estimated dev completion in about 2 weeks. 
    • There were questions around performance considerations and running under scale. 
• Discussion Topics
  • Airflow 3 adoption blockers
    • Vikram started the discussion with the commonly raised blockers that he had seen or been asked about:
    • Changes in imports: Vikram said that there seemed to be a significant lack of awareness of the upgrade utilities and the upgrade guide. Pavan chimed in saying that he had seen this too. There was some discussion about how to highlight this better on the docs site or in the UI, without a clear resolution. 
    • Missing UI elements such as UI filters: Brent chimed in saying that these have been identified as Github issues and are in the process of getting resolved. Brent also asked for community help here. 
    • Database size causing long migration times, due to long database upgrade times: Again, something covered in the upgrade guide, with the recommendation to clean up the database before upgrade. 
    • Vikram also said that he has been using the "affected_version:3.0" issue tag to track adoption issues and then with priority. 
    • The ask for the community is to "self-assign" adoption issues before the next meeting.
• Action Items
  • Pierre: Continue React plugin dev tools development
  • Brent: React package requirements optimization after vacation
  • Jarek: Investigate file description issue workarounds for pre-3.13
  • Dennis and Ramit: Finalize deadline alerts implementation
  • Kaxil, et al: Continue TaskSDK / shared code changes
  • Vikram: Clean up the Airflow 3.1 wiki structure and add Augusto Hidalgo to discuss Benchmark updates to next dev call. 
  • Community: Create, tag, and self-assign (if possible) Airflow 3 adoption issues before next dev call

31 July 2025

Proposed Agenda