Page History

...

Attendees

Summary

• Development Updates and Presentations:
  
  • AIP-72 Task Execution Interface aka Task SDK update (Ash, Kaxil)
    • Kaxil gave a detailed overview of the progress on this including the following highlights.
    • Secrets masker merged. This code is now shared between Task SDK and Airflow core
    • A couple of major PRs are landing this week. Includes: TP's DAG model removal and Kaxil's operator defaults schema changes (expect temporary breakage)
    • Schema defaults implementation is now in place with an updated contract between server/client with precedence hierarchy for defaults
    • A lot of the imports have now been cleaned up. It is 70% complete (12/17 issues merged), led by Ephraim.
    • Handling of the Configuration code in progress, with Amogh moving config to shared library (complex due to widespread usage)
  • AIP-86 Deadline Alerts update (Dennis, Ramit)
    • Dennis gave an overview of the progress, following up from the email to the dev list saying that the Async callbacks are working and will ship in 3.1 as planned. 
    • However, "Sync callbacks" will be delayed to 3.2. 
    • Dennis raised the question about whether the sync callbacks should wait for AIP-92 proposal implementation, since there is a possible conflict on callback architecture with that proposal. 
    • The consensus with the team at the dev call was that Deadline Alert Sync callbacks should continue with the existing design as proposed, rather than waiting for AIP-92. 
    • The action item was for Dennis to update the AIP (for Deadline Alerts) with this current status plus design update, and follow up with a lazy consensus email to the dev list. 
  • Airflow 3.1 test plan update (Rahul)
    • Rahul walked through an update of the 3.1 test plan and said testing was in progress with regression tests running on main and also with feature validation of the scoped AIPs on main. 
    • Rahul also highlighted that there were many open issues which needed to be triaged for the 3.1 milestone and that he was working on Amogh on testing for the import modifications. 
    • Rahul then handed off to Shahar for internationalizations, by saying that the translation owners were taking responsibility for testing their respective translations
  • Language translations update (Shahar)
    • Shahar said that there were would a English freeze a week early to enable translation completion before release
    • Additionally, there would be pre-commit check as enforcement to prevent English changes during the freeze
    • At this point, 4 translations needed committer sponsors (Catalan, Hungarian, Italian, and Portuguese) and the goal was to have 100% translation coverage before 3.1 release. 

...

Attendees

Summary

• Development Updates and Presentations:
  • Airflow 3.1 test plan update (Rahul)
    • Rahul shared that the overall testing status was positive across most components. 
    • He shared that the Task SDK regression testing had passed without import issues, the UI filters implementation were tested for audit log and XComm filters, and that the "Human in the loop" feature testing was on track. 
    • Rahul also said that AIP-81 testing ran into authentication issues with token/username-password. Bugra shared that the issue is being investigated and that the current suspicion is that it is environment related. 
    • Rahul was concerned about "UI Plugins" requiring retesting after recent changes by Brent and Pierre. 
    • Rahul also asked Dennis to update the testing status for "Deadline Alerts".
  • Airflow 3.1 beta release update (Kaxil)
    • Kaxil shared that the Airflow 3.1 beta version released earlier in the week did include the two major PRs highlighted in the last dev call i.e. TP's serialized DAG implementation changes and Kaxil's changes for defaults. 
    • As a result of the above, the server-side code now only uses serialized DAGs and Task Groups are decoupled between server and SDK. 
    • Also, migration testing from 2.11 to 3.1 beta has successfully been tested. 
  • Deadline Alerts (Dennis)
    • Dennis shared that async support was being added to Notifiers.
    • He also said that UI aspects were deferred to post 3.1 and that synchronous callbacks were in development, targeted for Airflow 3.2
  • Language translations update (Shahar)
    • Shahar shared that the language translations are at 97% coverage overall and we have been able to add new translations including Catalan and Simplified Chinese. Portuguese and Italian are still in PR, but may still be able to make it before release
    • And that the the translation freeze has been extended until the 3.1 release to get to 100% coverage
    • Amazing work here, I am astonished that we are now up to 14 language translations! 
• Discussion topics:
  • UI plugins recommendation (Pierre)
    • Pierre raised a discussion topic around marking React app plugins as experimental. 
    • Pierre said that the edge provider plugin work by Jens revealed needs for optimization and would probably require changes in build tools and dependency sharing. This would probably only affect very complex UI plugins, but it would be safer at this time to mark this part of the UI plugins work as "experimental". 
    • Based on this recommendation by Pierre and Brent, the team agreed with this recommendation. 
  • Airflow CTL update (Bugra)
    • Bugra said that the first beta was released for "airflow ctl" and that testing was ongoing with limited community interaction to date. 
    • Bugra said that a follow-on release candidate is planned on this, rather than a second beta. 
    • Following on, Bugra created a new AIP to decouple remote commands from the Airflow core CLI, which included a transition approach. 
    • Vikram, Jarek, and Kaxil raised concerns about existing automation dependencies based on the preexisting airflow CLI commands. 
    • Vikram recommended that Bugra delay asking for a vote on this AIP until after the 3.1 release, so that the team has the cognitive capacity to process the changes. 
• Release preparation and related action items:
  • Kaxil asked for the AIP (feature) owners to review the Airflow 3.1 draft release notes. Specifically including:
    • Human in the loop (Wei, et al)
    • Deadline alerts documentation (Dennis / Rajesh):
      • Mention of the async vs. sync differences and what's coming in 3.1 vs. later
      • Migration guide additions to explain the transition from SLA callbacks (in 2.x) to Deadline Alerts in 3.1 
    • UI React plugins documentation (PIerre / Brent):
      • Including what is experimental vs. not
  • Rahul asked Bugra to get back on the "airflow ctl" bugfix and testing
  • Rahul asked Dennis to update the Deadline Alerts testing
  • Vikram asked the team to review and cleanup the priority blocker issues (issues with label priority:upgrade_to_airflow3) and the 3.1 blocking issues
  • Elad asked Dennis to confirm what was needed for Notifier provider release in conjunction with the 3.1 release

...

Attendees

Summary

• Development Updates and Presentations:
  • Airflow 3.1.1 patch release update (Kaxil Naik)
    • Kaxil outlined that the 3.1.1 RC2 was cut earlier in the day and he expected 3.1.1 to be generally available on Monday next week.
    • Kaxil also said that he planned to cut the 3.1.2 RC1 before the end of October. Kaxil will be taking leave for a few weeks after that and Ephraim will be cutting the following patch releases. 
  • Airflow 2.11.1 patch release update (Jarek Potiuk)
    • Jarek spoke that this was primarily driven by a need to fix a key CVE and that he had to change a key dependency, but the tests were now working
    • Jarek said this would be a combined Airflow 2.11.1 and Fab provider 1.5.4 release
    • RC1 is planned for community testing, since this addresses multiple CVEs in dependencies. 
    • Jens offered to help with testing support. 
• Discussion topics 
  • Feedback from users about project at Airflow Summit
    • Vikram asked the team to share the feedback they had heard from the Summit regarding any Airflow 3 adoption challenges or critical issues
    • Amogh went first saying that he was folks had raised issues with niche use cases - primarily when they were leveraging the Airflow meta database directly for temporary data storage and therefore the lack of direct DB access was a blocker in Airflow 3 adoption. 
    • Vikram spoke about how we had told users in the past that they could work around this by explicitly defining a database connection to the Postgres database and using it explicitly, rather than implicitly through the Airflow internal interfaces, but also added that he wasn't sure if it was documented anywhere. Amogh said that he wasn't aware of that expressed solution and would create a Github issue to make sure it was documented. There was a larger discussion around this, including that users could leverage the published APIs (going through the API server) for this capability now. 
    • Dheeraj spoke about the issues his team at Qualcomm had faced in their upgrade. This was following up on his earlier email to the dev list and said that they had faced the issue above as well and had worked around it as expressed above. 
    • Dheeraj also raised the issue that the import changes were annoying, and that the Qualcomm team was not aware of the Ruff based utilities for upgrades. There was significant discussion around this as well and if this was a point in time issue with the early 3.0 builds. Ash asked Dheeraj if he could retry this with Airflow 3.1, since there was so much compatibility work done to keep the imports working and Dheeraj agreed to get back on this.
    • Dheeraj also said that there were several UI challenges around performance and a couple around navigation. Specifically, slow to load large DAGs, to list lots of DAGs, and log scrolling for large files. Also, viewing the UI from a mobile phone was problematic in 3 as compared to 2.x.  Brent and Pierre said that there were performance improvements in 3.1.1, but work was in flight around this. 
    • Dheeraj also said that very large DAG files i.e. with 23,000+ tasks were choking the DAG file processor. Kaxil asked Dheeraj to open a Github issue around this. 
  • Devlist followup on focusing on adoption issues 
    • Vikram asked the team if there were any objections to the proposal shared on the devlist to devote the next month i.e. mid-Oct to mid-Nov towards focusing on Airflow 3 adoption issues instead of significant new development. There were no objections to this.
    • Vikram then summarized that he would add this focus area to the Airflow wiki and that we could release a set of patch releases over the next four weeks to address these issues. 
    • Vikram also said that the logical timing for Airflow 3.2 would be to start development in earnest in mid-Nov, with a target release of March 2026, accounting for the holidays in late November and late December. There was general consensus. 
  • Why more providers needed ? Unknown User (potiuk) - deferred because of lack of time

...

Attendees

Summary

• Catch-up on action items from last call:
  • DAG import issues (Dheeraj)
    • Dheeraj said that he had re-tested the upgrade process and that the RUFF based utilities had significantly improved DAG compatibility from Airflow 2 to Airflow 3, when run with autofix, with over 50% of all the DAGs successfully parsed with Airflow 3, without needing manual changes. 
    • Dheeraj went to to say that the remaining issues requiring manual fixes were with: 
      • airflow.utils days_ago method,
      • DB create session no longer being available, because of direct database access removal
      • Simple HTTP Operator deprecation and Bash Operator being moved 
    • Dheeraj's summary was that the migration timeline after using the utilities would be about 3-4 days to achieve 80-90% DAG compatibility
    • The only remaining issue in his mind was UI performance, where it seemed that there was a noticeable slowdown as compared to Airflow 2.x
    • This report raised a fair amount of questions and discussion in the meeting itself. It was very helpful for the rest of the team to hear Dheeraj's feedback!

...

Attendees

Summary

• Catch-up on action items from last call:
  • Swim lane issue / PR tracking update (Unknown User (vikramkoka) )
    • Vikram share shared the output from the automated script he had created for tracking all open issues and PRs.
    • Using the 3.0 release date as a starting point, there are 308 open issues and 325 open PRs. 
    • Core Airflow had about 200 open issues out of which the UI and API swim lane had a third of all open issues and PRs.
    • Helm Chart had 18 open issues and 14 open PRs. 
    • Dev tools (including CI) had a surprisingly high number (34) of open PRs. 
    • Providers had about a quarter of the overall, but not yet classified in detail, because of a script bug.
    • Vikram also added that he would be updating the 3.x page with the updated swim lane information including metrics, and the swim lane leads. 
    • Vikram asked for volunteers to fill out the swim lane leads for areas which are backlogged.
    • Jens said that he would take the lead on Helm Chart, after chatting with Jed since Jed was now back from leave. 
    • Jarek suggested that "draft PRs" be excluded from the counts. 
    • Shahar asked about more detailed labels and leveraging the issue bot, but Vikram said that the issue bot was currently often wrong, requiring manual re-tagging. 
• Development Updates and Presentations:
  • Airflow 3.1.x patch release update (Unknown User (ephraimanierobi) )
    • Ephraim said that Core Airflow 3.1.3 was released last week
    • Ephraim added that 3.1.4 was planned for the first week of December, adding that it was originally targeted for a week earlier, but wanted to avoid Good Friday week. 
    • Jarek added that this was likely the last release before Christmas / New Year. 
    • Bugra added that airflowctl patch release dependencies were being resolved and that a new release candidate was being created. 
  • UI performance issues (Unknown User (pierrejeambrun)  / Unknown User (bbovenzi) )
    • Pierre said that the N+1 query issues were resolved and that the fixes were released as part of 3.1.3 
    • Pierre went on to say that the next priorities included adding missing indexes for sorting/filtering and that these needed to be configurable since the criteria could be different across user organizations. 
    • Brent added that additional work would be needed for View specific enhancements, such as Grid View performance enhancements. 
    • There was also a question raised about documenting guidance for the deployment of API servers for scale. 
  • AI-generated PR management strategy 
    • Vikram asked the question for the UI team about how many of the generated PRs (since the UI had the largest number of PR contributions) were AI-generated and if there were any quality issues
    • Brent responded that the PRs ranged from AI-assisted good PRs to very low-effort PRs which hadn't had enough thought behind them. 
    • Pierre said that his approach was to match the review effort to the contribution effort i.e. spend very little time reviewing low-effort PRs and just give high level feedback for obviously AI-generated PRs, so that time wasn't wasted.
    • Overall the team felt that they were spending a disproportionate amount of time reviewing poor quality PRs. 
    • Jarek said that ASF suggests (but does not mandate) disclosure of LLM usage in contributions
    • This led to a discussion on needing to update the contributing guide to reflect expectations on AI usage in the PR creation process. Ash took that action item to update the contributing guide. 
• Discussion topics:
  
  • Why more providers needed ? Unknown User (potiuk)
    • Jarek raised the challenge of increasing provider breadth, by saying that despite incoming requests for additional providers and people willing to contribute providers, we were limited by maintainer capacity. Jarek said that he had collaborated on a proposal with Vikram and others and would like to see this proposal move forward.
    • Vikram described the proposal for new providers as follows:
      • An Incubation step for new providers, then proceeding to "mainstream / production, and then to "attic / deprecation" if no longer being used
      • Contributors would need to take ownership responsibility for maintenance. 
      • Similar to the issue / PR metrics for core Airflow above, we would track and publish metrics around open issues, PR reviews, CVE fixes per provider 
      • Special case handling for mature, stable providers such as Slack. 
      • This process would be similar to how we handle translations with Shahar leading translations as a committer, but with individuals taking ownership for the actual creation and maintenance. 
      • We would continue to have only Apache Airflow committers as people who can merge, similar to translations. 
    • Radu (from Vespa) said that they wished to contribute an Airflow provider. Radu went on to say that the shared ownership model described earlier made sense would be mutually beneficial and that they would be incentivized to maintain their contribution, so that their users have a good experience
    • Vikram to update the AIP and send out a follow-up to the dev list
  • Multi-team to be aimed for 3.2 or 3.3 release (Vincent BECK )
    • Vincent wasn't at the meeting, but Niko chimed to say that they would like to target portions of the multi-team AIP for 3.2, with other elements following on in 3.3. 
    • Vikram asked about the status of the Deadline Alerts AIP, and Niko said that the remainder of the Notifier improvements were planned for 3.2
    • There will be a significant portion of the next dev call (on Dec 4th) dedicated to 3.2 scope planning. 

...