DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
Proposed Agenda
- Catch-up on action items from last call
- Airflow 3.0 minimum version to be Python 3.10 Unknown User (ash)
- Development Updates and Presentations (Please keep the updates to less than 10 minutes):
- Update on AIP-72 Task Execution Interface aka Task SDK
- Short circuit and Branch operators Unknown User (shahar)
- Test plan update Rahul Vats
- Milestone update (Unknown User (vikramkoka) )
- Feature completeness and road to RCs
- DevEx improvements as result of provider's distribution separation and devel
-common Unknown User (potiuk)
- Update on AIP-72 Task Execution Interface aka Task SDK
- Discussion topics:
- Packaging update - open items (Unknown User (ash) , Unknown User (jedcunningham) Unknown User (potiuk) )
- Edge provider naming issue (and PEP 752 as long term solution) Unknown User (potiuk)
- FAB next steps - Unknown User (potiuk)
Attendees
Vikram Koka
Ash Berlin-Taylor
Jens Scheffler
Rajesh Bishundeo
Eugene Kostielev
Kalyan Reddy Kasireddy
Pavan Kumar
Houda Cherkaoul
Vincent Beck
Ismael Cherkaoul
Ephraim Anierobi
Rahul Vats
Bugra Ozturk
Arunav Gupta
Pierre Jeambrun
Kenten Danas
Cedrik Neumann
Brent Bovenzi
Jed Cunningham
Shahar Epstein
Jarek Potiuk
Shubham Raj
Karthikeyan Singaravelan
Daniel Standish
Tamara Fingerlin
Abhishek Bhakat
Daniel Imberman
Aritra Basu
Daniel Hegberg
Vishnu Chilukoori
Shubham Mehta
Summary
- Follow-up on action items from the last call:
- Airflow 3.0 minimum version to be Python 3.10 Unknown User (ash)
- Moving to Python 3.10 is not a blocker.
- Ash said that he didn't have time to make this change in time for the 3.0 release.
- Therefore, decided to bump to after the 3.0 release and do this as part of the standard Python support cadence.
- Airflow 3.0 minimum version to be Python 3.10 Unknown User (ash)
- Development Updates and Presentations:
- Update on AIP-72 Task Execution Interface aka Task SDK
- Short circuit and Branch operators Unknown User (shahar)
- Shahar walked through the changes required for these operators.
- Specifically, the change from direct DB access to using the TaskSDK for updating XComs and the state of the task.
- Test plan update Rahul Vats
- Rahul walked through testing status with Beta1, including testing migrations (upgrades and downgrades).
- Rahul said that confidence level with respect to the various AIPs was increasing and that he expected the Grid of confidence by AIP to get a lot more colorful over the next week.
- He said that since the "DB isolation will be enforced starting Beta2", he expected more test DAGs to start failing.
- When asked about UI test automation, he said that the UI testing was manual and could not be automated yet because it was changing every day.
- The team was very appreciative of Rahul's update.
- Milestone update Unknown User (vikramkoka)
- Feature completeness and road to RCs
- Vikram shared the updated Airflow 3 dev milestones and said that he wanted the beta 4 release targeted for the week of March 17th, to be feature complete and the final beta release.
- Vikram asked for all open PRs to be completed, tested, and merged before beta 4. He urged contributors to ask for help if needed to meet this milestone.
- Vikram asked for patience for the community over the next few weeks, since open issues could not be allowed to linger and would be reassigned if the assignee was unresponsive.
- There was general consensus from the team.
- Vikram also brought up AIP-68 Extended Plugin interface for React Views, saying that this would need to be deferred to 3.1 because of timing. There was general agreement on this as well.
- DevEx improvements as result of provider's distribution separation and devel
-common Unknown User (potiuk)- Jarek shared a quick update on the recent DevEx improvements for Airflow development based on UV.
- He shared how this would make Provider development a lot easier than previously the case as well as benefiting TaskSDK development going forward.
- Update on AIP-72 Task Execution Interface aka Task SDK
- Discussion topics:
- Packaging update - open items (Unknown User (ash) , Unknown User (jedcunningham) Unknown User (potiuk) )
- A quick update shared was about making the Airflow repo a meta package, with Core Airflow becoming a project within the Airflow repo.
- Edge provider naming issue (and PEP 752 as long term solution) Unknown User (potiuk)
- Jarek brought up the issue that this name is already taken on the PyPI registry and that this is related to the PEP 752 discussion.
- When this PEP is implemented, ASF will take the "Apache" name, but this is still in progress with the POC to be implemented this week.
- The immediate action is to hopefully get this name (Edge) from the PyPI registry.
- FAB next steps - Unknown User (potiuk)
- Jarek confirmed that he is working on enabling Airflow to use FAB 5.0, since this is holding us back from upgrading SqlAlchemy.
- Jarek outlined multiple options:
- Option 1: Wait for 5.0, but that is 2 to 3 months
- Option 2: Take the needed parts of FAB and incorporate into the FAB provider; Then upgrade to SqlAlchemy 2.
- Option 3: Stay with SqlAlchemy 1 for now, but the security issues with it are problematic
- The team agreed that option 1 was a non-starter, and that we should on option 2, but be prepared to switch to option 3, if we run into issues with the SqlAlchemy upgrade being so close to release.
- Packaging update - open items (Unknown User (ash) , Unknown User (jedcunningham) Unknown User (potiuk) )