DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
Proposed Agenda
- Catch-up on action items from last call
- Vote for AIP-83 amendment (Unknown User (dstandish) , Unknown User (uranusjr) ) - in progress
- Development Updates and Presentations (Please keep the updates to less than 10 minutes):
- Update on AIP-82 External Event Driven Scheduling Vincent BECK
- Update on AIP-72 Task Execution Interface aka Task SDK Unknown User (ash) Unknown User (kaxilnaik) Unknown User (amoghdesai)
- Update on AIP-38 Modern Web Application Unknown User (bbovenzi)
- Update on Provider packages new structure Unknown User (potiuk)
- Discussion topics:
- AIP-81 CLI Discussion: Task Commands Remote vs. Local Buğra Öztürk (Doc link)
- Milestone and scope update Unknown User (vikramkoka)
- Airflow 2.11 release discussion update Unknown User (jscheffl) / Unknown User (kaxilnaik)
Attendees
Vincent Beck
Ash Berlin-Taylor
Jed Cunningham
Shahar Epstein
Daniel Standish
Jarek Potiuk
Ephraim Anierobi
Kalyan Reddy Kasireddy
Jens Scheffler
Elad Kalif
TP Chung
Rajesh Bishundeo
Vishnu Chilukoori
Bugra Ozturk
Shubham Raj
Karthikeyan Singaravelan
Pierre Jeambrun
Rahul Vats
Amogh Rajesh Desai
Augusto
Pavan Kumar
Tamara Fingerlin
Daniel Hegberg
Niko
Brent Bovenzi
Rafal
Michal Modras
Ankit Chaurasia
Shubham Mehta
Summary
- Follow-up on action items from the last call:
- AIP-83 amendment aka Data completeness discussion follow-up Unknown User (dstandish) Unknown User (uranusjr)
- Daniel Standish confirmed that the vote had been started on dev list and encouraged the team to respond.
- AIP-83 amendment aka Data completeness discussion follow-up Unknown User (dstandish) Unknown User (uranusjr)
- Development Updates:
- Update on AIP-82 External Event Driven Scheduling Vincent BECK
- Vincent demonstrated the implementation of the AssetWatcher concept built on top of FileOperator, which enables a DAG to be triggered based on the arrival of a file in an external blob storage.
- Vincent also did a second demo based on watching for two different files and that the DAG would be triggered based on either file arriving in blob storage.
- Vincent also showed a DAG which could be triggered based on a message arriving in an SQS queue.
- Vincent highlighted that DAG authors need to be careful on what triggers to use for invoking DAGs, since it is possible to have additional DAG runs because of persistent events. There was discussion around this and the agreement was to take that discussion offline.
- Update on AIP-72 Task Execution Interface aka Task SDK Unknown User (ash) Unknown User (kaxilnaik) Unknown User (amoghdesai)
- Ash highlighted that the team was chipping away on the work. He highlighted that Mapped Tasks was close to completely working based on the TaskSDK.
- He said that once Mapped Tasks was done, Celery and Kubernetes executors would be moved over to use the TaskSDK.
- He also said that after Mapped Tasks was done, the Trigger changes was the last "big feature" to be done and then there was going to be the long tail of fixes.
- Update on AIP-38 Modern Web Application Pierre Jeambrun Unknown User (bbovenzi)
- Pierre demonstrated the triggering and clearing of DAG runs and Tasks.
- Pierre said that Brent is working on the Grid view and it isn't merged yet, but close
- Pierre said that after the alpha, we would be able to use the new API exclusively and then we could delete the old UI and API.
- Pierre also said that they were making progress and no particular blockers.
- Brent demonstrated the GridView and said how they were looking forward to getting feedback as part of the Alphas.
- Update on Provider packages new structure Unknown User (potiuk)
- Jarek did a quick walk through about moving the providers to the new structure as a prelude to the new packaging work upcoming in February.
- He said that he would ask for help from the larger team on this work, since it could not be 100% automated. He said that his script automated 90+% of the work, but needed human intervention for the last little bit. He walked the team through a quick run of this script to illustrate.
- Update on AIP-82 External Event Driven Scheduling Vincent BECK
- Discussion Topics
- Milestone update and progress Unknown User (vikramkoka) (actually Unknown User (ash) )
- Ash walked through the overall progress update, since Vikram was out sick and couldn't make the call.
- Ash gave a quick overview of where we are based on the dev milestone wiki page. He highlighted that we are shooting for getting the first alpha builds out next week.
- Elad asked for clarification about the "feature freeze" beyond the alpha milestone. Ash took the action item to post a message to the dev list about it.
- Rajesh asked about where we are on the timeline and different AIP owners went through their progress updates as compared to the published timeline
- Jed, Vincent, Pierre, Vincent, Bugra, and Daniel Standish all went through their AIPs and expressed confidence in the schedule as they saw it.
- Ash summarized the discussion to express that the first alpha would have rough edges, but that the team felt positive about the progress and where we are at this point.
- AIP-81 CLI Discussion: Task Commands Remote vs. Local Buğra Öztürk (Doc link)
- Bugra wanted to have a discussion around some of the CLI Task commands which would not be possible through the Task API.
- Because of a lack of time, the proposal was to have that discussion async.
- Bugra suggested reviewing the doc also for a discussion around local vs. remote commands.
- Airflow 2.11 release discussions Unknown User (jscheffl) (Unknown User (potiuk) stepped in because of bad Wifi connection for Jens)
- Jed said that Utkarsh is planning to cut 2.10.5 next week.
- Topics raised demanded/needed for 2.11
airflow config lintcommand (but still can be added in 2.10-line or independent as well, no big driver)- Migration tooling (which is still WIP)
- No other things raised
- Generation notion: Not critical at the moment. At least nobody raised a deadline or critical feature (yet)
- Milestone update and progress Unknown User (vikramkoka) (actually Unknown User (ash) )
- Action items on/before next dev call:
- Ash to send out an email to the dev list regarding "feature freeze" guidelines, both to developers and committers about "merging PRs".
- Team to review Bugra's doc around CLI changes