Proposed Agenda

Attendees

  • Vincent Beck

  • Ash Berlin-Taylor

  • Jed Cunningham

  • Shahar Epstein

  • Daniel Standish

  • Jarek Potiuk

  • Ephraim Anierobi

  • Kalyan Reddy Kasireddy

  • Jens Scheffler

  • Elad Kalif

  • TP Chung

  • Rajesh Bishundeo

  • Vishnu Chilukoori

  • Bugra Ozturk

  • Shubham Raj

  • Karthikeyan Singaravelan

  • Pierre Jeambrun

  • Rahul Vats

  • Amogh Rajesh Desai

  • Augusto

  • Pavan Kumar

  • Tamara Fingerlin

  • Daniel Hegberg

  • Niko

  • Brent Bovenzi

  • Rafal

  • Michal Modras

  • Ankit Chaurasia

  • Shubham Mehta

Summary

  • Follow-up on action items from the last call:
  • Development Updates:
    • Update on AIP-82 External Event Driven Scheduling Vincent BECK 
      • Vincent demonstrated the implementation of the AssetWatcher concept built on top of FileOperator, which enables a DAG to be triggered based on the arrival of a file in an external blob storage. 
      • Vincent also did a second demo based on watching for two different files and that the DAG would be triggered based on either file arriving in blob storage. 
      • Vincent also showed a DAG which could be triggered based on a message arriving in an SQS queue. 
      • Vincent highlighted that DAG authors need to be careful on what triggers to use for invoking DAGs, since it is possible to have additional DAG runs because of persistent events. There was discussion around this and the agreement was to take that discussion offline. 
    • Update on AIP-72 Task Execution Interface aka Task SDK Unknown User (ash) Unknown User (kaxilnaik) Unknown User (amoghdesai) 
      • Ash highlighted that the team was chipping away on the work. He highlighted that Mapped Tasks was close to completely working based on the TaskSDK. 
      • He said that once Mapped Tasks was done, Celery and Kubernetes executors would be moved over to use the TaskSDK. 
      • He also said that after Mapped Tasks was done, the Trigger changes was the last "big feature" to be done and then there was going to be the long tail of fixes. 
    • Update on AIP-38 Modern Web Application Pierre Jeambrun Unknown User (bbovenzi) 
      • Pierre demonstrated the triggering and clearing of DAG runs and Tasks. 
      • Pierre said that Brent is working on the Grid view and it isn't merged yet, but close
      • Pierre said that after the alpha, we would be able to use the new API exclusively and then we could delete the old UI and API. 
      • Pierre also said that they were making progress and no particular blockers. 
      • Brent demonstrated the GridView and said how they were looking forward to getting feedback as part of the Alphas. 
    • Update on Provider packages new structure Unknown User (potiuk)  
      • Jarek did a quick walk through about moving the providers to the new structure as a prelude to the new packaging work upcoming in February. 
      • He said that he would ask for help from the larger team on this work, since it could not be 100% automated. He said that his script automated 90+% of the work,  but needed human intervention for the last little bit. He walked the team through a quick run of this script to illustrate. 
  • Discussion Topics
    • Milestone update and progress Unknown User (vikramkoka)  (actually Unknown User (ash)
      • Ash walked through the overall progress update, since Vikram was out sick and couldn't make the call. 
      • Ash gave a quick overview of where we are based on the dev milestone wiki page. He highlighted that we are shooting for getting the first alpha builds out next week.
      • Elad asked for clarification about the "feature freeze" beyond the alpha milestone. Ash took the action item to post a message to the dev list about it. 
      • Rajesh asked about where we are on the timeline and different AIP owners went through their progress updates as compared to the published timeline
      • Jed, Vincent, Pierre, Vincent, Bugra, and Daniel Standish all went through their AIPs and expressed confidence in the schedule as they saw it. 
      • Ash summarized the discussion to express that the first alpha would have rough edges, but that the team felt positive about the progress and where we are at this point.
    • AIP-81  CLI Discussion: Task Commands Remote vs. Local Buğra Öztürk  (Doc link)
      • Bugra wanted to have a discussion around some of the CLI Task commands which would not be possible through the Task API. 
      • Because of a lack of time, the proposal was to have that discussion async. 
      • Bugra suggested reviewing the doc also for a discussion around local vs. remote commands.
    • Airflow 2.11 release discussions Unknown User (jscheffl) (Unknown User (potiuk) stepped in because of bad Wifi connection for Jens)
      • Jed said that Utkarsh is planning to cut 2.10.5 next week.
      • Topics raised demanded/needed for 2.11
        • airflow config lint command (but still can be added in 2.10-line or independent as well, no big driver)
        • Migration tooling (which is still WIP)
        • No other things raised
      • Generation notion: Not critical at the moment. At least nobody raised a deadline or critical feature (yet)
  • Action items on/before next dev call:
    • Ash to send out an email to the dev list regarding "feature freeze" guidelines, both to developers and committers about "merging PRs". 
    • Team to review Bugra's doc around CLI changes
  • No labels