Agenda

• Catch-up on action items from last call
  • AIP-83 amendment aka Data completeness discussion follow-up Unknown User (dstandish) Unknown User (uranusjr) 
    • Docs to read beforehand:  Amendment to AIP-83 and Option 2 clarification 
• Development Updates and Presentations (Please keep the updates to less than 10 minutes):
  • Update on AIP-82 External Event Driven Scheduling Vincent BECK – defer to next dev call
  • Update on AIP-66 DAG Bundles & Parsing Unknown User (jedcunningham) 
  • Update on AIP-81 Enhanced security of CLI by integration with API Buğra Öztürk 
• Discussion topics:
  • Doc build process Unknown User (potiuk) 
  • When do we plan to release 2.11? And are all needed features available by then? Unknown User (jscheffl) 

Attendees

• Vikram Koka

• Ash Berlin-Taylor

• Jed Cunningham

• Shahar Epstein

• Daniel Standish

• Jarek Potiuk

• Ephraim Anierobi

• Kaxil Naik

• Jens Scheffler

• Elad Kalif

• TP Chung

• Vincent Beck

• Vishnu Chilukoori

• Bugra Ozturk

• Indraneel

• Karthikeyan Singaravelan

• Constance Martineau

• Rahul Vats

• Wilson Wang

• Phani Kumar

• Albert Okiri

• Tamara Fingerlin

• Pavan Kumar

• Michal Modras

• Ryan Hatter

• Arunav Gupta

• Kenten Danas

• Shubham Mehta

• Regan

Summary

• Follow-up on action items from the last call:
  • AIP-83 amendment aka Data completeness discussion follow-up Unknown User (dstandish) Unknown User (uranusjr)
    • The docs shared earlier were:  Amendment to AIP-83 and Option 2 clarification 
    • Daniel and TP walked through the detailed "option 2 clarification" document above to address the questions under discussion
    • At a high level, there was agreement about the overall approach. There were some details to be worked through.
    • Elad suggested a deadline to close the discussion, so that this could be done in time for release. Vikram agreed and recommended that Daniel / TP start an email voting thread, so this could be closed by Thursday of next week. 
    • The team also expressed appreciation to Daniel and TP for their perseverance in working through this change, since this had generated huge debate.
• Development Updates:
  • Update on AIP-66 DAG Bundles & Parsing Unknown User (jedcunningham) 
    • Jed demonstrated DAG bundles based on a Github repo, then DAG versioning and DAG fetching. 
    • He started a DAG run based on DAG version 1, updated the DAG, and then showed how the currently running DAG continued to execute subsequent tasks based on version 1 of the DAG, rather than the latest version.
    • Jed highlighted that this work had been a team effort with Ephraim and Daniel Standish.
    • DAG versionsing is a long requested feature within Airflow and it was great to see the working demo.
  • Update on AIP-81 Enhanced security of CLI by integration with API Buğra Öztürk
    • Bugra presented an update on the progress to date on the CLI
    • Bugra demonstrated the progress on making the CLI use the API for enhanced security, along with the command for connection configuration, as well as the authentication integration in flight. 
    • This AIP has been flying under the radar for a bit, so it was a treat to see the wonderful progress here.
• Discussion topics:
  • Doc build process Unknown User (potiuk) 
    • Jarek brought up the concern that the Airflow doc builds were taking 30 minutes now, from around 5 minutes a couple of years ago. He, Ash, Elad, Jed, and Kaxil all highlighted that this was a ticking time bomb. 
    • As a quick analysis, there seemed to be three different areas which needed to be addressed here:
      • The doc build technology which has not been updated in years
      • Process changes around Airflow user documentation including retention, versioning, and search engine optimization
      • Doc structure changes as needed for the upcoming Airflow 3 release.
    • Jarek also brought up the point that the typical folks who have been patching this are all very busy with Airflow 3 development and will be till release. Constance volunteered to take on the third item identified above. 
    • We could not finish the discussion because of time constraints, but the takeaway was to start an async discussion thread around this topic. 
  • When do we plan to release 2.11? And are all needed features available by then? Unknown User (jscheffl) 
    • Jens brought up the topic of Airflow 2.11, saying that we had discussed cutting the Airflow 2.11 release when we started working on Airflow 3 and if that was still in the plan.
    • The Airflow 2.11 discussion was initially triggered by the forward compatibility proposal with AIP-80. That approach had been changed before AIP approval and subsequently AIP-80 itself had been deferred to post-3.0. Since then, the Airflow 2.11 release faded away from the discussion.
    • Kaxil and Jens took the action item to find the issues tagged for Airflow 2.11 and to start an async discussion around the pros/cons of that release. 
• Action items on/before next dev call:
  • Unknown User (dstandish)  / Unknown User (uranusjr)  to start a voting thread for the AIP-83 amendment. 
  • Unknown User (potiuk)  to start an async discussion around the doc build issues
  • Unknown User (jscheffl)  / Unknown User (kaxilnaik)  to start an async discussion around the pros/cons for the 2.11 release