Page History

...

• Catch-up on action items from last call:
  • Airflow 3.1 Release plan and wiki update (Vikram)
    • Vikram walked the team through the Airflow 3.x wiki page.
    • He said that this was primarily focused on Airflow 3.1 at this time, but intended to be expanded for 3.2 and more in the future
    • Vikram outlined the target dates for release branch, release candidates, with a goal of cutting the GA release the week of September 15th, i.e. about 2 weeks before Airflow Summit. 
    • He said that all the work streams showed good progress, with several getting close to completion. He outlined that there was still a lot of work still to be done on the Task SDK split, but it was making good progress.
    • Vikram said that he was concerned about the UI filters stream of work and requested community help on that front. 
    • Jens asked about the Example DAGs work and Vikram confirmed that he had deliberately not included that in the 3.1 scope because of timing and was targeting that work for the 3.2 release.
• Development Updates
  • AIP-90 Human in the loop update (Wei)
    • Wei had sent a recorded demo of the working functionality from a dev branch.
    • It was amazingly comprehensive, covering the entry operator, the approval operator, and the branch operator.
    • The demo covered both the UI implementation and the API endpoints which could be used for programmatic interaction.
    • Jens confirmed that he had reviewed the PRs and suggested that the AIP was ready for broader UX feedback. 
    • Vikram mentioned that he had spoken with a large Airflow user who was currently using Airflow → Temporal → Airflow as a workaround and was excited for this functionality to eliminate the external dependency.
  • AIP-72 Task Execution Interface aka Task SDK update (Ash and Amogh)
    • Ash covered how the team was working on making the TaskSDK distribution independent of Airflow Core, as well as fixing upgrade blockers such as edge cases and virtual environment connection access. 
    • He mentioned that the key goals for 3.1 were:
      • TaskSDK installable without Airflow Core, and
      • Go SDK complete and ready for use
    • Amogh spoke about the test framework which has now been merged. He said that this now spins up a real Airflow instance to test SDK interface integration with the API server. 
  • Benchmark (AIP-59 Performance tests framework) update (Augusto Hidalgo)
    • Eugene said that Augusto was on vacation, but that the testing framework was running on both Airflow 2.x and Airflow 3, producing results.
    • Vikram asked if Augusto could present at the next dev call or alternatively to share the results async before the next dev call. 
  • AIP-81 Enhanced security in CLI update - airflowctl release planning (Bugra)
    • Bugra covered the detailed test cases created for validating compatibility of airflowctl with Airflow Core 3.1 
    • Bugra said that most bugs were fixed, with one or two open issues. 
    • The proposed release timelines was to start RCs by mid-August, with a target release date of September 1st. 
    • Ash asked the question about compatibility with Airflow 3.0.3 vs. Airflow 3.1, at which point Jarek suggested integrating tests into CI similar to Amogh's work above. 
    • Bugra said the that intention was to deprecate some of the existing CLI commands in the future, to avoid duplication with those commands in airflowctl. 
  • AIP-86 Deadline Alerts update (Dennis, Rajesh, and Ramit)
    • Rajesh said that the team expected to be code complete (for async callbacks) this week. 
    • Dennis confirmed that no UI requirements existed for this AIP, since it was purely a code invocation with notifications. 
    • Dennis confirmed that async callbacks running in the Triggerer were working now and attempted a demo, but the demo gods interfered. 
    • Ramit said that Sync callbacks to be run in the workers would be coming next week. This is similar to the 2.x SLA callback functionality and would be the easiest upgrade path. 
• Discussion Topics
  • Migration blocker issues (Elad)
    • Elad added a label for user-reported upgrade blockers, so that we could easily identify issues preventing 2.x → 3.x upgrades. 
    • Jarek asked the community to help review and give inputs, even if they were unable to code a fix. 
• Action Items
  • Pierre: Request help from the community to work on UI filters (on the dev list and on slack)
  • Augusto Hidalgo: Share performance test results async when back from vacation
  • Dennis / Ramit: Demo the sync callback functionality in the next dev call
  • Community: Review Human-in-the-loop UX and provide feedback
  • Community: Contribute to UI filters work and upgrade blocking issues 

...

• Development Updates and Presentations:
  • Airflow 3.1.x patch release update (Ephraim Anierobi)
    • Ephraim said that 3.1.2 had been released successfully.
    • Jarek reported that there was one issue reported right after about disappearing logs which may be critical and require a follow-on patch release. 
    • Rahul chimed in to say that this log issue was reproducible and a fix had also been identified and tested. 
    • There was agreement that this may require a 3.1.3 very soon, instead of waiting for the 2 week release cycle.
    • This is currently scheduled for this week and added to the Airflow 3.x wiki page
  • UI performance issues (Pierre Jeambrun)
    • Pierre reported that a number of N+1 query problems had been identified, were being resolved, and guard rails being put in place. The root cause was serialization layer lazy-loading relationships in loops. 
    • Pierre also referenced an issue that had identified missing indexes as a source of slowness and this was being resolved by new index creation. Vikram raised his concern that new index creation could cause issues in the "DB migration" part of an Airflow upgrade. Ash concurred with the concern and proposed a solution to make index creation part of the API server/ Scheduler startup rather than as part of the migration. 
    • Brent added that the Grid view performance remains challenging and that additional optimization work was being planned after the N+1 fixes were complete. 
    • There was also discussion about FastAPI configuration changes because of scaling differences from the Flask approach. This triggered a need for updating the documentation to recommend scaling approach recommendations. 
  • Auth issues (Vincent Beck)
    • Vincent reported that issues related to Auth were being resolved and that he had taken this on at Vikram's request. 
  • Expanding Task SDK Integration test framework with more tests (Amogh)
    • There was a quick ask for help from Amogh requesting community contributions to the Task SDK integration test framework. 
    • Amogh said that the complexity was higher than previous efforts and may require a SIG on slack for coordination. 
• Discussion topics:
  • Issue triage process (Vikram)
    • Vikram followed up on his email summary of issues sent to the dev list earlier, saying that the "needs triage" label was applied to 73 of the 284 open issues related to Airflow 3. And that this still seemed to be applied even after a PR had been created to address the issue. 
    • Jarek chimed in to say that this was unintentional and that at least he himself often forgot to assign or remove labels during the review process. 
    • Vikram proposed adoption of issues into logical swim lanes, with volunteer owners for those lanes such as:
      • Auth issues: Vincent leading
      • UI / API issues: Pierre and Brent leading
      • Data aware scheduling: TP and Wei leading
      • Edge Worker: Jens leading
    • There was some discussion around this with senior contributors such as Ash, saying that they look at everything, not based on individual areas. However, Vincent and Jens chimed in saying that this would be useful for them to focus attention. Brent chimed in saying that sometimes issues were mislabeled with the UI tag, but that was solvable by reassigning the UI-labelled issues post-initial triage. 
    • An aspiring contributor commented that these swim lane labels would also be useful for issues tagged with "Good first issue", so that they could pick something to work on based on their own skills and interests. 
    • Elad pointed out that this needs to be tried out in practice and if it works, also applied to PRs, since there are many PRs sitting waiting for approval. At this point, we have hit a record of 343 open PRs in the project! 
  • Thoughts on how to document DB access options in Airflow 3 upgrade docs (Amogh)
    • Amogh said that the Database access options topic had raised a lot of discussion on the PR. 
    • As a result, Amogh started a dev list discussion and was looking for input. Based on that, a lazy consensus would be started middle of next week. 

...

• Catch-up on action items from last call:
  • Airflow 3.1 Release plan and wiki update (Vikram)
    • Vikram walked the team through the Airflow 3.x wiki page.
    • He said that this was primarily focused on Airflow 3.1 at this time, but intended to be expanded for 3.2 and more in the future
    • Vikram outlined the target dates for release branch, release candidates, with a goal of cutting the GA release the week of September 15th, i.e. about 2 weeks before Airflow Summit. 
    • He said that all the work streams showed good progress, with several getting close to completion. He outlined that there was still a lot of work still to be done on the Task SDK split, but it was making good progress.
    • Vikram said that he was concerned about the UI filters stream of work and requested community help on that front. 
    • Jens asked about the Example DAGs work and Vikram confirmed that he had deliberately not included that in the 3.1 scope because of timing and was targeting that work for the 3.2 release.
• Development Updates
  • AIP-90 Human in the loop update (Wei)
    • Wei had sent a recorded demo of the working functionality from a dev branch.
    • It was amazingly comprehensive, covering the entry operator, the approval operator, and the branch operator.
    • The demo covered both the UI implementation and the API endpoints which could be used for programmatic interaction.
    • Jens confirmed that he had reviewed the PRs and suggested that the AIP was ready for broader UX feedback. 
    • Vikram mentioned that he had spoken with a large Airflow user who was currently using Airflow → Temporal → Airflow as a workaround and was excited for this functionality to eliminate the external dependency.
  • AIP-72 Task Execution Interface aka Task SDK update (Ash and Amogh)
    • Ash covered how the team was working on making the TaskSDK distribution independent of Airflow Core, as well as fixing upgrade blockers such as edge cases and virtual environment connection access. 
    • He mentioned that the key goals for 3.1 were:
      • TaskSDK installable without Airflow Core, and
      • Go SDK complete and ready for use
    • Amogh spoke about the test framework which has now been merged. He said that this now spins up a real Airflow instance to test SDK interface integration with the API server. 
  • Benchmark (AIP-59 Performance tests framework) update (Augusto Hidalgo)
    • Eugene said that Augusto was on vacation, but that the testing framework was running on both Airflow 2.x and Airflow 3, producing results.
    • Vikram asked if Augusto could present at the next dev call or alternatively to share the results async before the next dev call. 
  • AIP-81 Enhanced security in CLI update - airflowctl release planning (Bugra)
    • Bugra covered the detailed test cases created for validating compatibility of airflowctl with Airflow Core 3.1 
    • Bugra said that most bugs were fixed, with one or two open issues. 
    • The proposed release timelines was to start RCs by mid-August, with a target release date of September 1st. 
    • Ash asked the question about compatibility with Airflow 3.0.3 vs. Airflow 3.1, at which point Jarek suggested integrating tests into CI similar to Amogh's work above. 
    • Bugra said the that intention was to deprecate some of the existing CLI commands in the future, to avoid duplication with those commands in airflowctl. 
  • AIP-86 Deadline Alerts update (Dennis, Rajesh, and Ramit)
    • Rajesh said that the team expected to be code complete (for async callbacks) this week. 
    • Dennis confirmed that no UI requirements existed for this AIP, since it was purely a code invocation with notifications. 
    • Dennis confirmed that async callbacks running in the Triggerer were working now and attempted a demo, but the demo gods interfered. 
    • Ramit said that Sync callbacks to be run in the workers would be coming next week. This is similar to the 2.x SLA callback functionality and would be the easiest upgrade path. 
• Discussion Topics
  • Migration blocker issues (Elad)
    • Elad added a label for user-reported upgrade blockers, so that we could easily identify issues preventing 2.x → 3.x upgrades. 
    • Jarek asked the community to help review and give inputs, even if they were unable to code a fix. 
• Action Items
  • Pierre: Request help from the community to work on UI filters (on the dev list and on slack)
  • Augusto Hidalgo: Share performance test results async when back from vacation
  • Dennis / Ramit: Demo the sync callback functionality in the next dev call
  • Community: Review Human-in-the-loop UX and provide feedback
  • Community: Contribute to UI filters work and upgrade blocking issues 

...

• Development Updates and Presentations:
  • Airflow 3.1.x patch release update (Ephraim Anierobi)
    • Ephraim said that 3.1.2 had been released successfully.
    • Jarek reported that there was one issue reported right after about disappearing logs which may be critical and require a follow-on patch release. 
    • Rahul chimed in to say that this log issue was reproducible and a fix had also been identified and tested. 
    • There was agreement that this may require a 3.1.3 very soon, instead of waiting for the 2 week release cycle.
    • This is currently scheduled for this week and added to the Airflow 3.x wiki page
  • UI performance issues (Pierre Jeambrun)
    • Pierre reported that a number of N+1 query problems had been identified, were being resolved, and guard rails being put in place. The root cause was serialization layer lazy-loading relationships in loops. 
    • Pierre also referenced an issue that had identified missing indexes as a source of slowness and this was being resolved by new index creation. Vikram raised his concern that new index creation could cause issues in the "DB migration" part of an Airflow upgrade. Ash concurred with the concern and proposed a solution to make index creation part of the API server/ Scheduler startup rather than as part of the migration. 
    • Brent added that the Grid view performance remains challenging and that additional optimization work was being planned after the N+1 fixes were complete. 
    • There was also discussion about FastAPI configuration changes because of scaling differences from the Flask approach. This triggered a need for updating the documentation to recommend scaling approach recommendations. 
  • Auth issues (Vincent Beck)
    • Vincent reported that issues related to Auth were being resolved and that he had taken this on at Vikram's request. 
  • Expanding Task SDK Integration test framework with more tests (Amogh)
    • There was a quick ask for help from Amogh requesting community contributions to the Task SDK integration test framework. 
    • Amogh said that the complexity was higher than previous efforts and may require a SIG on slack for coordination. 
• Discussion topics:
  • Issue triage process (Vikram)
    • Vikram followed up on his email summary of issues sent to the dev list earlier, saying that the "needs triage" label was applied to 73 of the 284 open issues related to Airflow 3. And that this still seemed to be applied even after a PR had been created to address the issue. 
    • Jarek chimed in to say that this was unintentional and that at least he himself often forgot to assign or remove labels during the review process. 
    • Vikram proposed adoption of issues into logical swim lanes, with volunteer owners for those lanes such as:
      • Auth issues: Vincent leading
      • UI / API issues: Pierre and Brent leading
      • Data aware scheduling: TP and Wei leading
      • Edge Worker: Jens leading
    • There was some discussion around this with senior contributors such as Ash, saying that they look at everything, not based on individual areas. However, Vincent and Jens chimed in saying that this would be useful for them to focus attention. Brent chimed in saying that sometimes issues were mislabeled with the UI tag, but that was solvable by reassigning the UI-labelled issues post-initial triage. 
    • An aspiring contributor commented that these swim lane labels would also be useful for issues tagged with "Good first issue", so that they could pick something to work on based on their own skills and interests. 
    • Elad pointed out that this needs to be tried out in practice and if it works, also applied to PRs, since there are many PRs sitting waiting for approval. At this point, we have hit a record of 343 open PRs in the project! 
  • Thoughts on how to document DB access options in Airflow 3 upgrade docs (Amogh)
    • Amogh said that the Database access options topic had raised a lot of discussion on the PR. 
    • As a result, Amogh started a dev list discussion and was looking for input. Based on that, a lazy consensus would be started middle of next week. 

...