[WIP] AIP-85 DAG importer

Status

Motivation

What problem does it solve? / Why is it needed?

This AIP is a scoped-down, modified part of bigger initiative, which aims to establish extendable DAG parsing controls within Airflow. We’ve strategically divided  it into 2 independent parts to better manage complexity: while this AIP focuses on the implementation of the DAG importer interface, the subsequent AIP covers the overarching architectural framework for parsing controls with addition of interactive DAG processor (AIP-110). This modular approach allows us to implement the core importer mechanism faster and independently of the broader architectural updates.

The primary motivation for this AIP is:

• Many organizations are building customized workflow orchestration solutions on top of Airflow, often preferring formats like YAML to define their DAGs. Currently, these solutions require significant workarounds because Airflow is primarily designed to parse DAGs exclusively from Python files, even when the underlying generation process is a simple, standardized transformation.
• While AIP-72 successfully decoupled task definitions from Python, DAG structural definitions remain strictly bound to the Python language. This limitation is increasingly relevant as industry adoption of non-Python DAG generation tools (such as DAG Factory/Orchestration Pipelines/YAML and serverless configurations like AWS MWAA) grows. Allowing these alternative formats to be first-class citizens in Apache Airflow would:
• Eliminate the need for complex, brittle hacks to support external DAG definitions.
• Enable native support for diverse DAG generation sources.
• Align Airflow’s architecture with modern, multi-format workflow requirements.

Considerations

What change do you propose to make?

The high-level architecture centers on a modular registry-based approach, decoupling DAG discovery from parsing. This enables support for diverse DAG formats beyond Python files, such as YAML or serialized formats, by delegating parsing to specialized, pluggable importer implementations. The architecture for the proposal entails the following flow:

The design of the solution consists of three integral changes: 

• Airflow Core
• Airflow Configuration
• Airflow Providers

Proposal for Airflow Core changes

Introduce extensible architecture centered around two primary entities within the Airflow core: AbstractDagImporter and DagImporterRegistry.

AbstractDagImporter

This is the base interface responsible for the discovery, validation, and construction of DAG objects. It serves as an adapter layer that decouples the DAG definition source from the Airflow runtime. Importers are designed to be stateless or trivially constructible, ensuring they do not introduce complexity when managed across IPC boundaries in Airflow’s multi-process environment.

Proposed Interface

• @classmethod supported_extensions() -> list[str]: Defines the file extensions this importer can parse.
• import_file(file_path: str, *, bundle_path: str, bundle_name: str, safe_mode: bool) -> DagImportResult: Parses the target file and returns a DagImportResult containing parsed DAGs, import errors, and warnings.
• can_handle(file_path: str) -> bool: Determines if the importer is suitable for a given path, defaulting to a check against supported extensions.
• list_dag_files(dir: str, safe_mode: bool) -> Iterator[str]: Traverses the directory to identify files compatible with this importer.

The DagImportResult dataclass will standardize output across different formats:

class DagImportResult:
    file_path: str
    dags: list[DAG]
    errors: list[DagImportError]
    skipped_files: list[str]
    warnings: list[DagImportWarning]

DagImporterRegistry

The DagImporterRegistry acts as a centralized service locator and orchestrator. It maintains a 1:1 mapping between DAG Bundles and their configured importers. When the DagBag attempts to discover DAGs, it queries this registry to identify the correct importer for a given file path.

Proposed Interface

• register(importer: AbstractDagImporter, extensions: list[str]) -> None: Registers the DAG importer to handle the specific extensions.
• get_importer(file_path) -> AbstractDagImporter | None: Gets the registered importer for the path.
• can_handle(file_path) -> bool: Checks if there is a registered importer that can handle this path.
• supported_extensions() -> list[str]: Lists the extensions for which there is a supported registered AbstractDagImporter.
• list_dag_files(dir, safe_mode) -> list[str]: Lists the dag files in the directory via registered DAG importers. 

DagBag backwards compatibility

To maintain strict backward compatibility across the Airflow, particularly for thousands of existing unit tests and internal utilities that instantiate DagBag, the DagImporterRegistry will be integrated using an optional dependency injection pattern. The DagBag.__init__ method accepts an optional importer_registry argument. When omitted, the constructor automatically falls back to invoking the global, default-initialized DagImporterRegistry singleton instance, which arrives pre-loaded with the PythonDagImporter, ensuring that legacy tests and standard Python-based DAG environments parse exactly as they did in legacy versions, requiring zero code modifications from end-users or provider developers.

def __new__(cls) -> DagImporterRegistry:
        with cls._lock:
            if cls._instance is None:
                cls._instance = super().__new__(cls)
                cls._instance._importers = {}
                cls._instance._register_default_importers()
        return cls._instance

def _register_default_importers(self) -> None:
        from airflow.dag_processing.importers.python_importer import PythonDagImporter

        self.register(PythonDagImporter())

Implemented foundation [as of Jan 2026]

The foundation for the above entities / changes was laid in the following PR.

Archive importers (specifically ZipImporter)

Currently, the existing PythonDagImporter owns both the .py and .zip extensions. This design has two critical limitations:

• Python DAG parsing and ZIP archive extraction logic are tightly coupled inside a single importer.
• Users cannot package non-Python DAG formats (like YAML DAGs or custom domain-specific language DAGs) within a .zip archive. The current system is locked into assuming that any .zip file exclusively contains Python modules.

To solve these limitations, we propose introducing a generic ZipImporter (and a wider pattern for nested/hierarchical importers). The ZipImporter is responsible only for parsing and extracting .zip archives, then delegating the discovery and parsing of files contained inside the archive to specialized, configurable "internal" importers (format specified in Airflow Configuration changes below).

Proposed design:

1. Existing AbstractDagImporter classes expect a standard, absolute file path on the local file system for their import_file() invocation. To maintain complete backward compatibility and prevent rewriting existing/custom importers to accept raw bytes or file streams, ZipImporter will follow an extraction-to-temp-dir execution flow.
2. When the DagBag parses a bundle and encounters a .zip file:
1. DagImporterRegistry routes the .zip path to ZipImporter
2. ZipImporter creates a secure, isolated temporary directory using tempfile.TemporaryDirectory()
3. It reads the ZIP table of contents
4. To protect against disk-space depletion, ZipImporter only extracts files whose suffixes match the keys in the configured internal_importers dictionary
3. To support cross-imports within the ZIP archive (e.g., a dag.py importing a helpers.py utility contained in the same ZIP), the temporary directory is temporarily injected into sys.path.
4. ZipImporter runs file discovery inside the temporary directory. For each discovered file, it selects the configured internal importer (e.g., invoking PythonDagImporter for .py files and YamlDagImporter for .yaml files), and passes the file's extracted path on the local filesystem to the internal importer's import_file() method.
5. As a result, importer collects the DagImportResult from all internal importer runs and translates the temporary absolute file locations in the results back to relative zip-internal file paths (e.g. my_archive.zip:dags/dag_1.py) to ensure user-facing error logs, warnings, and DAG metadata references remain readable.
6. As a cleanup step, the temporary directory is fully deleted, and sys.path entries are cleaned up.

Proposal for Airflow Configuration changes

This section defines the mapping between DAG bundles and their corresponding importer logic. The configuration can be part of [dag_processor]dag_bundle_config_list, for example:

[
  {
  	"name": "dags-folder",
  	"classpath": "airflow.dag_processing.bundles.local.LocalDagBundle",
    "kwargs" {}
    "importers": {
    	"py": {
        	"classpath": "airflow.dag_processing.importers.python_importer.PythonImporter"
        },
        "zip": {
        	"classpath": "airflow.dag_processing.importers.archive.ZipImporter"
            "kwargs": {
            	"internal_importers": {
                	"py": {
                    	"classpath": "airflow.dag_processing.importers.python_importer.PythonImporter"
                    }
            }
        }
    }
  }
]

Defining this mapping within the bundle configuration allows Airflow to maintain strict isolation between different sources/bundles. This structure ensures that parsing logic is coupled with the bundle's lifecycle, simplifying dependency management and reducing the risk of configuration being applied across diverse environments.

Design considerations:

• Resolution order: Importers are resolved based on file extension matching. If multiple importers support the same extension, the importer will be overridden, and, based on the current implementation, will log a warning.
• Composition: The archive importer pattern (as shown above) demonstrates how importers can be nested. This recursive resolution allows for complex bundle structures (e.g., an archive containing multiple DAG definition types). 

Proposal for Airflow Provider changes

To truly leverage the decoupled parsing architecture of AbstractDagImporter, Airflow's provider ecosystem must be extended to support a new provider entity type: DAG Importers. This allows third-party packages to author, distribute, and register custom DAG parsers natively.

The technical details of defining, registering, and utilizing a custom importer involve 2 steps: defining the custom importer interface and registering the importer in provider metadata.

Defining a custom importer interface

Users or provider developers must create a class that inherits from AbstractDagImporter and implements its core interface. The custom importer dictates how files are read, translated into Airflow DAG objects, and returned via the standardized DagImportResult dataclass. Structure:

from airflow.dag_processing.importers.base import AbstractDagImporter, DagImportResult

class NewDagImporter(AbstractDagImporter):

    @classmethod
    def supported_extensions(cls) -> list[str]:
        ...

    def can_handle(self, file_path: str) -> bool:
        ...

    def import_file(
        self, file_path: str, *, bundle_path: str, bundle_name: str, safe_mode: bool
    ) -> DagImportResult:
        ...

Registering the importer in provider metadata

To make the custom importer discoverable by Airflow environments, the provider must declare it in its provider.yaml file (or get_provider_info() dictionary). A new dag_importers key will be introduced to the provider schema:

"dag-importers": {
    "type": "array",
    "description": "List of custom DAG importer classes exposed by this provider.",
    "items": {
        "type": "object",
        "properties": {...},
    }
}

When Airflow initializes, the ProviderManager will scan installed packages for dag_importers and make their classpaths readily available for bundle configurations, similar to Hooks, Operators, and Triggers.

Considerations

Performance

To validate that architecture introduces no latency regressions, the following dimensions will be measured as part of the performance tests:

• Number of DAG files
• Number of DAGs in single file
• Number of accesses to other parse-time parameters

Given that the initialization of the DagBag with DagImporterRegistry and loading of importer classes, some small performance regression is anticipated. At the same time alternative parsers will provide the performance improvement in themselves (e.g. by avoiding unnecessary parsing) which can overweight any anticipated regression.

What defines this AIP as "done"?

This AIP will be considered “done” when the PR with necessary well-documented solution is created and the language-specific DAG importer providers can be registered.