Airflow 3.0 Development Milestones

The goal of this page is to document all the development milestones for the Airflow 3.0 release. This is intended to be a "living" document and reflect plans and progress during the development and test phase of the project. This is supplemental to the high level timelines in the main Airflow 3.0 page.

This was initially documented in preparation for the dev call on the 10th of October and reviewed as part of that dev call.

Date	Scope
Week of 24 Oct 2024	Completion AIP-74 Introducing Data Assets AIP-83 Remove Execution Date Unique Constraint from DAG Run AIP-79: Remove Flask AppBuilder as Core dependency AIP-65: Improve DAG history in UI Calculate / Store observed DAG version in DB Keep history of serialized DAG and code AIP-66: DAG Bundles & Parsing Initial DAG Bundle interface and git/local merged AIP-72 Task Execution Interface aka Task SDK POC of Task Execution with Core / Task SDK split POC of Remote Python Task Execution POC of non-Python language support (GO) AIP-69 Edge Executor (Initial Name: Remote Executor) (MVP on main / Rework on AIP-72 open) AIP-38 Modern Web Application Updated designs for Dashboard / Home page, DagsList, DAG, DagRun, Task, and TaskInstance pages WIP AIP-75 New Asset-Centric Syntax (Backend changes) AIP-78 Scheduler-managed backfill (Backend changes) AIP-72 Task Execution Interface aka Task SDK Split Core & Task SDK Articulate distribution strategy for Task SDK AIP-68 Extended Plugin Interface for React Views (Working POC) AIP-38 Modern Web Application DagsList, Dashboard (started, but will iterate) AIP-84 Internal API for UI Operations (some endpoints migrated) Unknowns (none) Won't be started AIP-76 Asset Partitions AIP-80 Explicit Template Fields in Operator Arguments AIP-81 Enhanced Security in CLI via Integration of API AIP-82 External event driven scheduling in Airflow
Week of 18 Nov 2024	Completion AIP-75 New Asset-Centric Syntax (Backend changes) AIP-78 Scheduler-managed backfill (Backend changes) AIP-65: Improve DAG history in UI (Backend changes complete) API endpoints support retrieving versioned history Finalized detection of observed DAG version AIP-66: DAG Bundles & Parsing Ability to configure DAG bundles via CLI/API Use DAG bundles in parsing loop AIP-72 Task Execution Interface aka Task SDK Split of Core and Task SDK Initial incarnation of Python execution based on split AIP-38 Modern Web Application DAG, DagRun and Task pages including DAG History (based on AIP-65) (Draft, but not final) WIP AIP-81 Enhanced Security in CLI via Integration of API Dynamic generation of API requests Central API communication mechanism Missing API Endpoints AIP-84 Internal API for UI Operations (most endpoints migrated) AIP-72 Task Execution Interface aka Task SDK Logging support for Tasks Finalize complex cases such as Dynamic Tasks for task execution support AIP-69 Edge Executor (Initial Name: Remote Executor) (Migrate to AIP-72) AIP-75 New Asset-Centric Syntax (Frontend changes) AIP-78 Scheduler-managed backfill (Frontend changes) AIP-76 Asset Partitions AIP-68 Extended Plugin Interface for React Views (Working POC with most core functions) Performance benchmark scenarios to be shared by end of Nov (loosely based on AIP-59 Performance tests framework) Unknowns ... Won't be started AIP-80 Explicit Template Fields in Operator Arguments
Week of 16 Dec 2024	Completion AIP-84 Internal API for UI Operations API fully migrated to FastAPI Functional and improved compared to legacy AIP-66: DAG Bundles & Parsing Complete support for DAG bundles in task execution Add support for manifest Add support for some parsing control in manifest AIP-72 Task Execution Interface aka Task SDK Complete Task Execution in python based on split Core / Task SDK Celery integration with TaskSDK AIP-82 External event driven scheduling in Airflow Depends on AIP-75 New Asset-Centric Syntax (above) AIP-69 Edge Executor (Initial Name: Remote Executor) (Code Freeze, stabilization on AIP-72) AIP-68 Extended Plugin Interface for React Views (Code Freeze if possible) AIP-38 Modern Web Application Assets and DAG dependencies (based on AIP-74 changes) Completed DAG, DagRun and Task pages including DAG History WIP AIP-72 Task Execution Interface aka Task SDK Integration with AIP-66 for DAG fetching AIP-38 Modern Web Application Backfill related changes (based on AIP-78 changes) Initial data collected to be done and shared for performance benchmark - by end of Dec (loosely based on AIP-59 Performance tests framework)
Week of 13 Jan 2025	Completion AIP-72 Task Execution Interface aka Task SDK Integration with AIP-66 for DAG fetching and versioning Callback integration Finalize TaskSDK distribution AIP-38 Modern Web Application Backfill related changes (based on AIP-78 changes) Asset changes finalized (based on AIPs 74 and 75)

Space shortcuts

Page tree