Page History

Status

Table of Contents

Table of Contents
maxLevel 4 exclude Table of Contents|Status

Motivation

Airflow, as a leading open-source platform for data engineering and orchestration, sits at the center of many production data platforms - coordinating , coordinating critical pipelines, integrations, and cross-team dependencies.

As AI-assisted development and operations become mainstream, Airflow users increasingly expect to see productivity gains for operational debugging and day-to-day troubleshooting: faster root-cause analysis, clearer next steps, and reduced dependence on tribal knowledge. Several vendors and large deployments are already experimenting with AI-assisted debugging for orchestration systems, indicating clear demand in the ecosystem.

When things go wrong, the cost is paid in on-call time, delayed pipelines, and repeated tribal-knowledge debugging across Dag runs, task instances, logs, configs, and permissions.

Today, teams that want AI-assisted debugging in Airflow create ad-hoc, private integrations , or rely on paid solutions. This creates duplicated effort, inconsistent security controls, and uneven user experience across deployments.

An official, opt-in plugin assistant provides a safe and consistent foundation for AI assistance: a standard UI entry point, a supported backend API surfaceinterface, and a controlled way to connect tools via MCP (Airflow API, logs, metadata, etc.) retrieve Airflow state so answers can be grounded in real system state. This proposal does not embed any specific model or vendor into core Airflow, but instead defines an extension point and UI surface that allows organizations to integrate AI capabilities according to their own policies.data.

Because Airflow already centralizes the key signals needed for debugging (Dag definitions, task metadata, run state, logs, configs, and RBAC), it is well positioned to benefit from “AI + tools” AI-assisted tooling to reduce time-to-diagnosis and improve operational reliability, without requiring Dag authors to change their code.

The design must respect existing RBAC boundaries and ensure that any tool access is scoped to the current user’s permissions, preserving Airflow’s security model and minimizing data exposure risks.

Making this official lets the community define best practices around security, auditability, and rollout, so organizations can adopt AI debugging responsibly rather than reinventing it.

Moreover, by defining clear and supported extension points for AI integrations, Airflow becomes an attractive platform for experimentation and contribution from developers working in AI, observability, and tooling, strengthening the community and accelerating innovation around the project.

Problem Definition

What problem does it solve?

• Today, troubleshooting Airflow issues often requires manually navigating between DAG definitions, task instances, scheduler/worker logs, configuration files, and external documentation. This diagnostic process can be slow and error-prone, particularly for new users or on-call responders who must piece together information from multiple sources.

• While Airflow’s plugin infrastructure and public APIs support custom integrations for monitoring, logging, or secret backends, there is no standardized, secure way to integrate conversational assistants or tooling that can interpret and correlate Airflow state across these artifacts. As a result, teams experimenting with AI-assisted tools often build ad-hoc solutions with inconsistent security controls and uneven user experience, leading to duplicated effort and uneven risk handling.

Why is it needed?

Airflow is increasingly used by a broad range of users, including data engineers, analytics teams, and platform operators - to build, schedule, and monitor complex data workflows. Troubleshooting failures and understanding pipeline behavior often requires manually piecing together information from multiple sources (Dag definitions, task instances, scheduler logs, configuration state, and documentation), which can be slow, error-prone, and difficult for newcomers or on-call responders to navigate effectively. Improved visibility and guidance can reduce mean-time-to-diagnosis and support burden.

Because actionable insights depend on live Airflow state - such as the current status of runs, task logs, and metadata - an assistant is only valuable if it can access and correlate that information through a standardized, secure interface. Existing plugin and integration mechanisms in Airflow support extensibility, but there is currently no standard way to integrate conversational assistants or tooling that can safely interpret and relate Airflow’s internal state in a consistent manner.

Standardizing the integration surface (plugin hooks plus a tool interface) reduces fragmentation, enables consistent security controls, and allows operators to adopt AI-assisted insights with predictable behavior across deployments. Making the capability opt-in and plugin-based aligns with Airflow’s extensibility philosophy and permits organizations that prohibit AI usage to opt out entirely.

Considerations

This proposal introduces a new integration surface between Airflow and external AI systems, expanding the system’s security and operational surface.

Key considerations include:

• Security surface expansion: The assistant may interact with external systems and process operational metadata. Strong guarantees around RBAC, redaction, and auditability are required.
• Operational complexity: Enabling the feature requires additional configuration and operational awareness from deployment managers.
• Opt-in nature: The feature must remain disabled by default and explicitly enabled by operators.
• Dependency on standardized tooling interfaces: State-backed assistance depends on a consistent and secure way to retrieve Airflow state.
• Maintenance overhead: The feature introduces additional long-term maintenance across UI, backend, and integration layers.

The design must strictly respect existing RBAC boundaries and ensure that the assistant never exceeds the permissions of the authenticated user.

What change do you propose to make?

This AIP proposes introducing an opt-in AI Assistant capability for Apache Airflow.

The assistant provides:

• A conversational interface within the Airflow UI.
• The ability to answer user questions about Airflow workflows.
• Responses that may be grounded in live Airflow state through controlled, read-only access.

Phase 1 is explicitly limited to read-only assistance and does not allow any modification of Airflow state.

Image Added

Image Added

Figure 1 - Illustrations of the Assistant UI (screenshots from an initial POC, final implementation may differ)

 to illustrate one possible approach to realizing this proposal.

That document is intended to support discussion and demonstrate feasibility; it does not prescribe the final design.

What problem does it solve?

Troubleshooting Airflow workflows currently requires manually navigating between multiple sources of information, including Dag definitions, task instances, logs, configuration, and documentation.

This process is:

• Time-consuming
• Error-prone
• Dependent on user experience and tribal knowledge

There is no standardized way to query and correlate this information through a single interface.

Why is it needed?

Users increasingly expect AI-assisted workflows to improve productivity in debugging and operations.

Without a standardized approach:

• Teams build ad-hoc integrations
• Security controls vary across implementations
• User experience is inconsistent

Providing an official, opt-in assistant:

• Reduces duplicated effort
• Establishes consistent security and audit practices
• Enables predictable and safe adoption of AI assistance

Are there any downsides to this change?

Yes:

• Introduces additional security considerations due to interaction with external systems.
• Adds operational complexity for deployments that enable the feature.
• Increases maintenance overhead for the project.
• Depends on evolving AI ecosystem components and integration surfaces.

Organizations with strict data handling requirements may choose not to enable this feature.

Which users are affected by the change?

• Deployment Managers: responsible for enabling, configuring, and operating the assistant.
• Dag authors & Operational Users: interact with the assistant to retrieve insights about workflows.

How are users affected by the change? (e.g. DB upgrade required?)

By default (feature disabled):

• No behavioral change.
• No additional configuration or services required.

When enabled:

• Deployment managers must configure and operate the assistant capability.
• Additional components may be required to support assistant functionality.
• The system’s security posture changes and must be managed accordingly.

The feature is opt-in and does not require changes to existing Dags or workflows.

What is the level of migration effort (manual and automated) needed for the users to adapt to the breaking changes? (especially in context of Airflow 3)

There are no required migrations or breaking changes for existing users when the feature is disabled (default).

The assistant is introduced as an opt-in capability, and therefore does not impact existing Dags, workflows, or deployments unless explicitly enabled.

When the feature is enabled:

• Deployment Managers must perform manual setup steps, such as enabling the feature and configuring required integrations.
• Additional metadata storage may be introduced for audit and operational purposes, which may require applying setup or migration steps as part of enabling the feature.
• No changes are required from Dag Authors: existing Dag definitions continue to work unchanged.

At this stage, migration effort is low and primarily operational, with no expected need for automated migration utilities, as the feature does not modify existing user code or behavior.

Future phases that introduce state-modifying capabilities may require additional migration considerations and would be addressed in separate AIPs.

Other considerations?

• Read-only retrieval of Airflow state (e.g., Dags, task instances, logs, selected metadata).
• Responses that may be grounded in retrieved system state.
• Optional - responses that do not rely on live system state (for examples, general questions about Airflow).

Out of scope

• Any state-modifying operations (create, update, delete).
• Triggering or retrying workflows or tasks.
• Automated remediation or execution.
• Privilege escalation beyond the authenticated user’s permissions.
• Autonomous or background actions.

Future phases may explore controlled, permissioned state-modifying capabilities under separate AIPs.

Constraints and guarantees

The following must always hold:

• The feature is disabled by default.
• The assistant operates in a strictly read-only mode in Phase 1.
• All access is enforced via existing RBAC mechanisms.
• The assistant must never exceed the permissions of the authenticated user.
• Sensitive data must be redacted before any external processing.
• All interactions must be auditable, with configurable retention and redaction.
• Failures must not impact scheduler, executor, or core Airflow functionality.

Behavioral expectations

• Users interact with the assistant via natural-language queries.
• Responses may incorporate Airflow state when relevant.
• The assistant must clearly distinguish between:
  • Responses based on live system state
  • General, non-grounded responses
• The assistant must not fabricate unavailable data and must communicate limitations clearly.

Security considerations

• All data access must respect Airflow’s RBAC model.
• The assistant must not introduce privilege escalation.
• Sensitive information must be protected through redaction and data minimization.
• Interactions must be logged for auditability and traceability.
• The assistant must remain operationally isolated from core scheduling and execution components.

What

Scope

This AIP defines the first phase of an AI Assistance capability for Airflow.

In Scope (Phase 1)

The initial implementation is limited to:

• A read-only assistant interface embedded in the Airflow UI.

• Retrieval and interpretation of Airflow state (e.g., Dag runs, task instances, logs, configuration metadata) through controlled tooling interfaces.

• Generation of advisory, informational responses based on current Airflow state.

• Enforcement of existing RBAC permissions and security boundaries.

• Structured audit logging of assistant interactions.

The assistant does not modify Airflow state in this phase.

Out of Scope

The following capabilities are explicitly out of scope for this AIP:

• Creation, modification, or deletion of Airflow objects.

• Triggering or retrying Dag runs or task instances.

• Automated remediation or execution of workflows.

• Privilege escalation beyond the requesting user’s permissions.

• Autonomous background execution or unsupervised actions.

Future Phases

Future enhancements may explore safe and permission-controlled state-modifying actions (e.g., create/update/delete operations). Such capabilities would require a separate AIP that defines:

• Explicit permission models

• Guardrails and confirmation mechanisms

• Additional audit and safety controls

• Operational impact analysis

No state-modifying capabilities are included in this proposal.

Proposed Solution

High-level overview

This AIP introduces an official, opt-in AI Assistance capability for Airflow that enables users to ask natural-language questions about their workflows and receive grounded answers based on real Airflow state.

At a high level, this change introduces the usage of the following components (only when the feature is enabled):

1. Frontend chatbot plugin - A UI extension embedded in the Airflow web interface (e.g., as a dedicated tab or panel) that accepts natural-language questions and displays contextual, read-only answers or insights (see illustration in Figure 1). 

1. Plugin backend - A plugin component that exposes secured API endpoints co-deployed with the Airflow instance, receives frontend requests, coordinates with a configurable assistant backend, and mediates access to Airflow signals and tools in an audited and controlled way.
2. MCP integration surface – A standardized interface, as defined and implemented in AIP-91, that allows the assistant backend to call controlled tools that retrieve Airflow information (for example, Dag list, Dag runs, task instances, logs, etc.) in a way that respects RBAC and allowlisting policies. This AIP consumes that interface but does not define or implement it.

The first phase of this capability focuses on read-only insights and explanations of Airflow state. Actions that modify state (create, update, delete) such as triggering runs, modifying tasks, or altering configurations are explicitly out of scope for the initial implementation and may be considered in a future phase once safe action patterns, permission controls, and guardrails have been established.

This design uses Airflow’s plugin infrastructure (custom UI and API extension points) to minimize core changes while providing a secure and extensible foundation for AI-assisted workflow insights.

Image Removed 

Behavioral Model

User Interaction Model

When enabled, the assistant appears as an embedded UI surface within the Airflow web interface (see Figure 1 for illustration). 

The assistant operates in a conversational model:

• Users submit natural-language queries.

• Responses are generated in context of live Airflow state.

• Each interaction is scoped to the current authenticated user.

Image Removed

Image Removed

Figure 1 - Illustrations of the Chatbot Plugin UI

Query Processing Semantics

For each user query (see Figure 2):

1. The request is associated with the currently authenticated Airflow user.

2. The system evaluates the user’s RBAC permissions.

3. The assistant may invoke read-only tools to retrieve relevant Airflow state.

4. Retrieved data is filtered according to the user’s permissions.

5. The assistant generates a response grounded in tool outputs.

6. The response is displayed in read-only format.

Image Removed

Figure 2 - Query processing semantics diagram

The assistant never bypasses permission checks or accesses data unavailable to the requesting user.

Tool Invocation & Grounding

The assistant may use controlled tools to access Airflow's API:

Image Removed

Tool invocation is:

• Explicit

• Allowlisted

• Read-only (Phase 1)

• Audited

Responses are based on actual system state rather than pure model inference whenever possible.

Error and Failure Handling

The assistant must gracefully handle:

• LLM provider timeouts

• Rate limits

• Tool invocation failures

• Permission denials

Failure states must:

• Not block the Airflow UI

• Not affect scheduler or task execution

• Display clear user-facing error messages

Image Removed

Audit and Accountability

Each interaction:

• Is associated with a user identity.

• Is recorded according to audit policy.

• Includes sufficient metadata for traceability.

Image Removed

Response Characteristics

Responses must:

• Be clearly marked as AI-generated.

• Indicate when external data was retrieved.

• Provide citations or references to Airflow objects when applicable (e.g., Dag ID, Dag Run ID).

• Avoid fabricating unavailable data.

If grounding fails or required tools are unavailable, the assistant must communicate this explicitly rather than hallucinate.

 Scope Boundaries (Reinforced)

• No create, update, or delete actions.

• No triggering of Dag runs.

• No retrying tasks.

• No background autonomous behavior.

• No privilege escalation.

 UI/UX Principles

The assistant UI:

• Follows existing Airflow UI conventions.

• Does not obscure core UI functionality.

• Provides indicators for backend & MCP connectivity.

• Clearly communicates rate limits and errors.

• Clearly distinguishes between facts retrieved from Airflow state and model-generated analysis or suggestions.

Technical Architecture

UI Plugin

Security model

Database Schema and Migration Scope

The assistant feature introduces additional metadata (e.g., structured audit records) that must be persisted in Airflow’s metadata database when the feature is enabled.

To preserve the opt-in nature of this proposal and avoid impacting deployments that choose not to use AI assistance, assistant-related database tables:

• Are scoped exclusively to the assistant feature.

• Are not created as part of the base Airflow schema.

• Are created via a plugin-scoped database migration that is applied only when the assistant feature is enabled or installed.

These tables are clearly namespaced (e.g., ai_assistant_*) to avoid ambiguity with core Airflow metadata models and to ensure clear ownership boundaries.

When the feature is disabled, no assistant-related tables are created and no schema changes are required. When the feature is enabled, administrators must apply the associated migration to create the required metadata tables (such as audit logging tables).

This approach ensures:

• True opt-in behavior with no schema impact on default installations.

• Clear separation between core Airflow metadata and assistant-specific metadata.

• Explicit lifecycle management for assistant-related persistence.

• Governance clarity regarding ownership and maintenance of assistant-specific schema.

Assistant-related tables must comply with configured redaction and retention policies and must not persist sensitive data beyond what is required for traceability and accountability.

Security and controls

This proposal introduces an optional feature that, by design, may cause outbound network egress when an organization chooses to integrate with an external LLM or tooling provider. Unmonitored outbound traffic can expose sensitive information to third-party services and introduce new vectors for data leakage and exfiltration. This behavior must be opt-in, and operators must be able to define clear permitted boundaries for what data may be shared externally (e.g., restricting to specific metadata fields). In air-gapped or restricted environments, the feature must be disableable without impacting core Airflow functionality.

Because the assistant may analyze operational signals such as Dag code, logs, configuration metadata, and state, it is inherently security-sensitive. Potential risks include unintended data exposure, prompt injection vulnerabilities in LLM-integrated systems, and privilege escalation via an assistant acting with user permissions. To mitigate these risks, the design should:

1. Explicitly prohibit transmission of secrets, credentials, and connection passwords
2. Enforce Airflow RBAC boundaries at every interaction in alignment with the existing security model
3. Provide audit trails for all assistant activity
4. support administrator controls for allowlisting specific fields and automatic redaction of sensitive values.

Operational reliability & Configurability

This feature adds a new integration surface area. Documentation should clearly describe provider-side limits and behaviours, and - where feasible - Airflow should expose corresponding configuration controls (such as client-side timeouts and retry policies) to help operators manage dependencies on external services and reduce the impact of provider rate limits on Airflow stability.

The initial implementation phase is an integration framework and a minimal assistant UX; it is not a fully autonomous expert system and should not be relied upon for automated remediation without explicit operator verification.

Packaging and Release Model

The assistant feature will be delivered as an official plugin maintained under the Apache Airflow project. The packaging and repository model must be explicitly defined before this AIP is considered complete.

The following alternatives are considered:

Option 1 — Included as Part of Airflow Core (Single Release Cycle)

The assistant plugin would live inside the apache/airflow repository and be released strictly as part of the standard Airflow release cycle. It would follow the same versioning, branching, and backport policies as core Airflow.

Pros

• Clear signal of first-class support and long-term commitment.

• Unified release, branching, and compatibility model.

• No need for a separate compatibility matrix.

• Simplified distribution (shipped with Airflow).

Cons

• Increases the maintenance and governance surface of core Airflow.

• Reduces flexibility for rapid iteration during early maturity.

• Couples the assistant’s evolution tightly to Airflow’s release cadence.

• May create perception that AI functionality is embedded in the base distribution, even if feature-gated and disabled by default.

Option 2 - Hosted in apache/airflow monorepo, but independently versioned

The assistant plugin would live within the apache/airflow repository (e.g., under a dedicated subdirectory), but would have its own independent versioning and release cadence, similar to how providers or other separately packaged components like task-sdk or airflowctl are handled.

Pros

• Clear association with the main project while maintaining release flexibility.

• Easier coordination with core changes compared to a fully separate repository.

• Allows independent iteration without waiting for full Airflow releases.

Cons

• Adds complexity to the release process within the monorepo.

• Requires clear tooling and policy for independent packaging and publishing.

• May blur boundaries between "main" and "adjacent" features.

• Still increases CI, testing, and repository complexity.

Option 3 - Separate Official Repository under the Apache Airflow Namespace (Recommended)

The assistant plugin would be maintained in a dedicated repository (e.g., apache/airflow-ai), versioned independently, and released separately from core Airflow.

Pros

• Strongest opt-in separation from core functionality.

• Maximum flexibility for rapid iteration and stabilization.

• Reduces maintenance and governance risk for the main repository.

• Clean architectural and ownership boundaries.

• Allows experimentation and API evolution during early maturity.

• Enables clearer positioning within the broader AI ecosystem, potentially attracting new contributors and users interested in AI-assisted workflow tooling.

Cons

• Requires explicit compatibility management with supported Airflow versions.

• Slightly more operational complexity for users installing the plugin.

• Requires coordination for cross-repository changes affecting internal APIs.

Recommendation

It is recommended that the assistant plugin initially be hosted in a separate official repository under the Apache Airflow namespace (Option 3), with independent versioning and a clearly documented compatibility matrix for supported Airflow 3.x releases.

This approach:

• Preserves the opt-in nature of the feature

• Minimizes risk to the main repository

• Enables faster iteration during early maturity

• Maintains clear governance boundaries

Consolidation into the main apache/airflow  repository may be considered in a future AIP if appropriate, but long-term independent maintenance under a dedicated repository is also a valid outcome.

The chosen packaging and release model must be finalized prior to marking this AIP as complete.

Downsides & Tradeoffs

Security surface expansion

Operational complexity

The assistant introduces additional configuration and runtime considerations, including external LLM provider management, rate limits, timeouts, and monitoring. While isolated from core scheduling and task execution, it adds components that deployment managers must configure and operate when the feature is enabled.

Maintenance burden

This feature increases long-term maintenance overhead across UI, backend, MCP integration, and provider compatibility. Ongoing testing of security boundaries, API stability, and cross-version compatibility will be required. Clear ownership of interface contracts is essential.

Dependency on MCP implementation (AIP-91)

The assistant depends on the MCP tooling surface defined in AIP-91. Changes or delays in that interface may impact implementation timelines or require adaptation. This introduces cross-AIP coordination requirements.

Impact Analysis

Which users are affected?

• Deployment Managers: Configure and enable/disable the feature; manage credentials and policies; deploy and operate the assistant plugin components and the MCP tooling service (as defined in AIP-91).
• Operational Users: Use the assistant within the Airflow UI to retrieve read-only insights about workflows.

Migration & Upgrade impact

By default (feature disabled)

• No behavioral change.
• No database upgrade required.
• No additional services.

When enabled (opt-in)

• Deployment managers must explicitly configure:

  • A feature enable flag (e.g., AIRFLOW__AI__ENABLED or equivalent plugin configuration).

  • LLM provider credentials and endpoint (e.g., API key, URL) stored via an Airflow Connection.

  • The MCP tooling service endpoint and credentials, according to AIP-91, which the plugin backend uses to securely retrieve Airflow state.

• Operational users:

  • When the assistant feature is enabled and configured, the plugin backend (the component that implements the assistant logic) will use the configured Connection to make outbound API requests to the selected LLM provider.

  • The MCP tooling service must be reachable from the plugin backend.

  • Enabling the feature changes the system’s security posture and requires documented guardrails.

  • Additional operational considerations:

    • A database migration must be applied to create required AI-related tables (e.g., audit logs).

    • The MCP tooling service must be reachable from the plugin backend.

    • Enabling the feature changes the system’s security posture and requires documented guardrails.

What is the level of migration effort (manual and automated) needed for the users to adapt to the breaking changes?

Similar to Airflow-core, breaking changes to database schemas will be managed automatically.

defines this AIP as "done"?

This AIP is considered complete when the following criteria have been satisfied for the first phase are met for Phase 1 (read-only AI-assisted insightsassistance):

Functional completeness

• A production-ready implementation suitable for real-world deployments of Airflow 3.x, under the defined scope (read-only AI assistance).
• The assistant UI is integrated into the Airflow web interface using standard plugin extension points and is An opt-in assistant capability is available and accessible only to authorized users.
• The assistant can accept questions via the plugin UI and return grounded answers based on live Airflow state.
• The implementation supports integration with at least one configurable LLM provider and defines an extensible abstraction layer to support additional LLM providers.
• The implementation integrates with the standardized tooling interface defined in AIP-91 (once available) to retrieve Airflow signals in a secure, read-only manner.
• The assistant UI follows established Airflow UI patterns and integrates consistently with existing navigation and permission models.

Security and controls

• Permission and RBAC boundaries are enforced for all assistant access.
• Administrators can configure allowlists and redaction policies to control what data is exposed externally.
• Assistant interactions are persistently recorded in a structured audit log (e.g., a dedicated database table), capturing sufficient context for traceability and accountability, in accordance with configured redaction and retention policies.

Configurability and operational behavior

• Feature gating and default-off behavior are implemented.
• Administrators can configure timeouts, error handling behavior, and integration controls.
• Failure scenarios (such as provider errors or rate limits) have defined behaviors and do not compromise core Airflow availability.

Quality assurance

• Relevant unit, integration, and security tests are implemented and passing (including evaluation of system prompts by AI using Airflow's AWS instance).
• Performance characteristics meet agreed-upon expectations and do not degrade core Airflow functionality.
• UI behavior and permission boundaries are covered by integration tests.
• The assistant UI provides clear feedback for loading, error, and rate-limit states.
• AI-generated responses are clearly identified as such to users.

Documentation

• User and administrator documentation exists that explains how to enable/disable the feature, control data exposure, and configure credentials securely.
• A basic troubleshooting guide is provided for common error classes and operational problems.

Distribution and Release Readiness

• The packaging, repository location, and release model for the assistant feature (e.g., inclusion in apache/airflow or a separate official repository under the Airflow project) are defined and approved by the PMC.

• Versioning and compatibility expectations with supported Airflow 3.x releases are documented.

• Ownership and maintenance responsibilities are clearly defined.

• Users can submit natural-language queries and receive responses related to Airflow workflows.
• Responses can incorporate relevant Airflow state where applicable.
• All interactions strictly respect RBAC and security constraints.
• The feature is disabled by default and fully configurable by deployment managers.
• Failures in the assistant do not impact core Airflow functionality.
• Auditability and data protection requirements are enforced.
• Documentation is provided for enabling, configuring, and operating the feature.

Once these criteria are satisfied, the feature can be considered ready for adoption in real-world deployments under the defined Phase 1 scopeOnce all of these criteria are met, the feature in its first phase (read-only AI assistance) can be considered done and ready for adoption with production usage expectations.