Proposed Agenda

Catch-up on action items from last call
- Wiki update for Airflow 3.3 (Unknown User (vikramkoka) )
Airflow 3.3 and related Development Updates
- AIP-103 update - Unknown User (vikramkoka) , Unknown User (amoghdesai)
- AIP-108 Java-SDK and Coordinator update - Unknown User (jasonliu)
- AIP-86 Deadline Alerts update - Sean Ghaeli, Unknown User (ferruzzi)
- AIP-67 Multi-team update - Niko, Vincent BECK
- TBD
Discussion topics:
- MCP (AIP-91) & AI Assistant (AIP-101) - High-Level introduction (Unknown User (shahar) )
- TBD

Attendees

Ash Berlin-Taylor
Jason Liu
Vincent Beck
Shahar Epstein
Sean Ghaeli
Aaron Chen
Christos Bisias
Phani Kumar
Rajeshwar Bishundeo
Amogh Rajesh Desai
David Blain
Ephraim Anierobi
Pratiksha Badheka
Jens Scheffler
Jed Cunningham
Shivam Rastogi
Bugra Ozturk
Constance Martineau
Jarek Potiuk
Dheeraj Turaga
Brent Bovenzi
Vikram Koka
Karthikeyan Singaravelan
Aritra Basu

Summary

Note: Vikram was traveling to the airport and joined late. Ash Berlin-Taylor ran the first part of the call on his behalf.
Catch-up on action items from last call:
- Airflow 3.3 wiki update (Vikram)
  - Vikram was not yet on the call at this point. Ash noted he could not speak to the specific wiki actions and moved on.
  - Bugra noted in chat that AIP-94 had been added to the wiki after the last call.
Airflow 3.3 Development Updates:
- AIP-103 Task State Management (Amogh Rajesh Desai)
  - Amogh shared that development on AIP-103 began last week, with three or four PRs already merged. Work completed so far includes the ORM and database layer, the execution API and configurations, and the context accessor — wiring up the Task SDK with the execution API and providing task state through the context object.
  - Amogh gave a live demo using a simple Python service simulating a paginated newsletter API with an artificial downtime endpoint. Two versions of a DAG were demonstrated: a classic stateless operator and a new stateful operator using task state for checkpointing.
  - In the stateless version, when the service went down mid-run and the task retried, it restarted from index 0, repeating all previously completed API calls. In the stateful version, the task stored its progress at the end of each batch, and on retry resumed from where it left off — in the demo, from index 26 rather than 0.
  - David Blain noted that this is particularly valuable because XCom is cleared on retry, whereas task state persists across retries. He highlighted a specific use case for dynamic task mapping where the new capability would allow tracking which mapped tasks have succeeded and which have not, without needing the deferred mechanism workaround he currently uses.
  - Vikram confirmed this use case is explicitly covered in the AIP.
  - Shahar raised the question of how the stateful nature of a DAG should be surfaced in the UI, suggesting clear indicators and potentially a panel showing current state. Amogh confirmed this is planned but not yet implemented.
  - Ash clarified that AIP-103 defines multiple state scopes. The current demo covers task instance scope, which is the foundational element for the retry use case. Asset scope is a separate scope being worked on by Jake in the context of event-driven scheduling. Other scopes are envisioned for future capabilities.
  - Vikram noted that the AIP is designed as a foundation for a broader set of use cases over time.
  - Shivam asked in chat whether the DAG scope state is stored in the Airflow database or could be stored externally such as in S3. This was noted for follow-up.
  - AIP-103 doc: https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-103%3A+Task+State+Management
- AIP-108 Java SDK and Coordinator update (Jason Liu)
  - Jason gave an update on progress since the last dev call, walking through the key changes rather than repeating the full demo.
  - The most significant change is the public interface for multi-language task support. The team has settled on using the existing queue argument as the identifier for routing tasks to the appropriate language coordinator, rather than exposing a new language parameter. This avoids any table schema changes and requires no new execution API endpoints. The corresponding configuration has been renamed accordingly.
  - A second change involves the introduction of a Task Instance Data Transfer Object, needed to expose queue information from the task instance to the Task SDK world without relying on the shared module.
  - The third and most significant change is that all coordinator implementations will now live in a new distribution rather than under a provider. The new distribution will be installed via apache-airflow-coordinator-java and imported via a lightweight import utility, without the provider manager overhead. Jarek welcomed this as a new pattern for Airflow extensions beyond providers, noting it could pave the way for similar separations in other areas such as log handling.
  - Jason also previewed the architecture for Go and other languages, noting that for Go and Rust the coordinator simply needs to launch a binary executable rather than a JVM. A draft Go SDK implementation is in progress and already able to serve DAG source code from the executable. The interface is designed to be generic enough for new languages to adapt quickly.
  - Shivam Rastogi shared in chat that he is testing a TypeScript/Node SDK this week and has opened a draft PR: #66289.
  - Jason offered to share slides with a Mermaid diagram detailing the coordinator flow for DAG parsing and task execution, for those who want to understand the architecture in depth.
  - There was a brief discussion around the term "coordinator" — Jens noted it could be confusing, though no better alternative was proposed. Jarek said the name is appropriate given the role will expand over time.
  - AIP-108 doc: https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-108+Java+Task+SDK+and+the+Language+Coordinator+Layer
- AIP-107 Direct Queueing from Triggerer (Jens Scheffler)
  - This was not on the original agenda but Jens raised it briefly at the start of the call.
  - Jens noted he had written AIP-107, which addresses the latency problem for tasks returning from the Triggerer to a worker. He asked for community feedback on the dev list, as the AIP had received no responses so far.
  - Ash said he had a question about it and would look at it. Jarek also said he would add it to his list.
  - The discussion touched on the current limitation that the Triggerer cannot write XCom directly without going back through the worker, and that enabling supervisor-style async support in the Triggerer may be the right path forward.
  - AIP-107 doc: https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-107+Direct+Queueing+from+Triggerer+-+or+other+means+to+prevent+Latency+for+Tasks+exiting+deferred+State
- AIP-86 Deadline Alerts update (Sean Ghaeli)
  - Sean gave a status update on the remaining scope for AIP-86, with the goal of completing everything for the 3.3 release.
  - The theme is closing the loop on all existing work rather than adding new features. The one item that may not make 3.3.0 is migrating existing callback functionality (on-success, on-failure callbacks) to the new deadline alerts framework. Everything else is on track.
  - Sean asked anyone with questions to reach out on Zoom or Slack.
  - AIP-86 doc: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=323488182
- AIP-67 Multi-team update (Vincent Beck)
  - Niko was not on the call. Vincent gave the update.
  - Vincent walked through the current work on asset event filtering for multi-team deployments. By default, a DAG scheduled on assets will only be triggered by asset events sent by the same team. Vincent is implementing a new allowed_producer_teams attribute that allows a DAG to explicitly opt in to receiving asset events from other teams.
  - A design question was raised by Ash around what happens when a DAG from a non-allowed team updates an asset: does the asset get updated but the DAG not triggered, or is the update itself rejected? Jarek clarified that the asset gets updated, but the trigger does not fire for teams that have not allowed that producer. This is consistent with the existing data model which tracks asset updates and DAG triggers as separate entities.
  - Ash noted this could create a confusing user experience where an asset event is visible in the UI but a DAG did not run, and emphasized the importance of making this clear in the UI. Jarek agreed this should be surfaced clearly.
  - Ash also raised an edge case around asset watchers, where two conflicting definitions from different teams on the same asset ID could cause unexpected behavior. Jarek acknowledged this is an existing issue that teams exacerbate, and agreed it needs to be addressed.
  - AIP-67 doc: https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components
Discussion Topics:
- MCP (AIP-91) and AI Assistant (AIP-101) introduction (Shahar Epstein)
  - Shahar gave an introduction and live demo of the work he is proposing, which he had also presented at the Airflow Town Hall the week before.
  - The demo showed a prototype AI assistant built into the Airflow UI, which can answer natural language questions about the user's Airflow environment by connecting to an Airflow MCP server and retrieving live state. The example shown was "how many DAGs do I have," which returned a live answer.
  - Shahar outlined the motivation: currently users must manually navigate between DAGs, runs, task instances, logs, configs, and documentation to troubleshoot issues. An AI assistant with live Airflow context would reduce time to diagnosis significantly.
  - He noted that Astronomer has already launched its own AI assistant, and that some teams are building their own third-party integrations. The question for the community is whether an open-source alternative under the Apache Software Foundation is warranted.
  - Shahar proposed a two-stage approach: first finalize and release a stable MCP implementation (AIP-91), then build the AI assistant (AIP-101) on top of it. He proposed forming a task force or SIG to drive this work, with discussion in the existing MCP Slack channel and potential periodic video calls.
  - Ash asked whether the AI assistant needs to be in Airflow Core or could be a plugin. Vikram raised a broader challenge: does any of this need to be in the Airflow repo at all, or is this better suited as an ecosystem project? He drew a parallel with DAG builders and tools like Kubernetes or Celery, which Airflow integrates with but does not own, and emphasized the maintainability burden of keeping AI tooling current given how rapidly the space is moving.
  - Jarek framed the question differently, saying the more important distinction is whether this is released by the Apache Software Foundation PMC, as that carries a trust and brand implication. He also noted the MCP security landscape is very active with many known vulnerabilities being discovered, and cautioned against moving too fast. He suggested learning from what others have got right and wrong before committing.
  - Jens said from a UX perspective he sees real value for non-power users, and suggested the AI assistant could be implemented as a plugin with a small extension point added to the UI sidebar, keeping the footprint in core minimal. He agreed the MCP question needs to be resolved first. He also noted that whatever is built should use a pluggable model rather than bundling a specific model, so users can bring their own.
  - Brent added in chat that beyond MCP, Airflow should consider a plugin registry to make plugins more discoverable rather than adding more to core.
  - The team agreed the MCP question is the foundational one. Shahar committed to developing a security-focused approach for the MCP and bringing it back to the group for discussion.
  - Jens closed with a joke: "The letter S in MCP stands for Security."
  - AIP-91 doc: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=364349979
  - AIP-101 doc: https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-101+Airflow+AI+Assistant