DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
Proposed Agenda
- Catch-up on action items from last call
- TBD
- Development Updates and Presentations (Please keep the updates to less than 10 minutes):
- AIP-81 CLI Authentication Design Update Buğra Öztürk
- AIP-72 Task Execution Interface aka Task SDK Update Unknown User (ash) Unknown User (kaxilnaik)
- Test plan update Rahul Vats
- TBD
- Discussion Topics:
- Packaging update Unknown User (potiuk)
- Cleanup of Airflow 3 open items in Github - raised by Unknown User (eladkal)
- airflow3.0:candidate with 26 open items
- airflow3.0:breaking with 10 open items
- TBD
Attendees
Vikram Koka
Jens Scheffler
Jarek Potiuk
Ash Berlin-Taylor
Rajesh Bishundeo
Ambika Garg
Jed Cunningham
Kenten Danas
Kalyan R
Amogh Rajesh Desai
Rahul Vats
Tamara Fingerlin
Shahar Epstein
Zach Gottesman
Pavan Kumar
Bugra Ozturk
Eugene Kostieiev
Ephraim Anierobi
Elad Kalif
Kaxil Naik
Daniel
Abhishek Bhakat
Vincent Beck
Aritra Basu
Kathikeyan Singaravelan
Brent Bovenzi
Ian Buss
Pierre Jeambrun
Dennis Ferruzzi
Shubham Raj
Vishnu Chilukoori
Cedrik Neumann
Summary
- Development Updates and Presentations
- AIP-81 CLI Authentication Design Update Buğra Öztürk Unknown User (jedcunningham)
- Bugra and Jed explained how AIP-81 would create a completely separate remote CLI package, which will be "airflow-ctl".
- For local execution, all of the CLI commands (which are DB backed) will just work.
- All of the merged code for remote CLI execution will be moved into airflow-ctl as part of beta4 and will also simplify packaging and release timelines.
- AIP-72 Task Execution Interface aka Task SDK Update Unknown User (ash) Unknown User (kaxilnaik) Unknown User (amoghdesai)
- Kaxil gave an update on the AIP-72 detailing that JWT Integration, Custom XCom backend, and Secrets backend work is all completed.
- He added that the Triggerer backend now works without going through the public API, to avoid all the authentication passing complexity.
- Kaxil said that the Task Notifiers is the one open item, which TP is currently working on to finish.
- Amogh added that the Secrets backend also enables a separate secrets backend for Workers as compared to the Secrets backend configured for the Scheduler.
- Ash added that each Task now has a JWT token that is associated with it, bound to a Task Try ID.
- Ash also added that the Execution API versioning was now merged for the Task Execution Interface and implemented Calendar versioning, based on Cadwyn.
- Ash added that this was only for the Task Execution Interface, not for the public API, nor for UI internal API.
- Shahar added that he had heard that some AWS provider system tests were breaking right and that he would be investigating.
- Test plan update Rahul Vats
- Rahul shared the updated Airflow 3 Test plan and walked through a quick progress update by AIP.
- Rahul asked everyone on the team to put on their testing hat, treat beta4 as being the internal testing vehicle, and to test it out, so that we can find more bugs before release.
- Rahul also added that AIP-81 needed to be tested for CLI regressions.
- AIP-81 CLI Authentication Design Update Buğra Öztürk Unknown User (jedcunningham)
- Discussion Topics:
- Packaging Update Jarek
- Jarek highlighted the big PR which introduces the "airflow-core" package within the main airflow repo.
- He highlighted how there is no longer an "airflow" folder in the main repo.
- Instead, there is "airflow-core" which has all the key items inside it including the pyproject home.
- The second thing we have is an airflow meta package which has all the configuration for builds and additional elements for packaging and distribution.
- Jarek said that this was still work in progress, with a few things to be fixed, but was mostly done.
- Jarek also spoke about how UV works with the new packaging structure and stressed how this simplifies Airflow development.
- Cleanup of Airflow 3 open items in Github Unknown User (eladkal)
- Elad asked all the AIP owners to review the list of Github issues originally targeted for Airflow 3 and still open
- Specifically, these are airflow3.0:candidate and airflow3.0:breaking
- Elad made the point that these should either be closed as having been done or moved to a subsequent release tag as part of the Airflow 3 RCs.
- Packaging Update Jarek
- Action items
- What's critical to finish up
- Jens brought this up in the context of the UI, if there are open critical items to be completed before 3.0 release
- Brent took the action item to highlight this async and shout out if anything needed help
- Change dev calls to be weekly until release
- Elad suggested that we make the dev calls weekly, since we are very close to release.
- Vikram to update the calendar invite to be weekly and to send out a reminder if a meeting is required that week.
- What's critical to finish up