Page History

The goal of this page is to document all the development milestones for the Airflow 3.0 release. This is intended to be a "living" document and reflect plans and progress during the development and test phase of the project.

This was initially documented in preparation for the dev call on the 10th of October and reviewed as part of that dev call.

Date

Scope

Week of 24 Oct 2024

Completion
- AIP-74 Introducing Data Assets
- AIP-83 Remove Execution Date Unique Constraint from DAG Run
- AIP-79: Remove Flask AppBuilder as Core dependency
- AIP-65: Improve DAG history in UI
  - Calculate / Store observed DAG version in DB
  - Keep history of serialized DAG and code
- AIP-66: DAG Bundles & Parsing
  - Initial DAG Bundle interface and git/local merged
- AIP-72 Task Execution Interface aka Task SDK
  - POC of Task Execution with Core / Task SDK split
  - POC of Remote Python Task Execution
  - POC of non-Python language support (GO)
WIP
- AIP-75 New Asset-Centric Syntax (Backend changes)
- AIP-78 Scheduler-managed backfill (Backend changes)
- AIP-72 Task Execution Interface aka Task SDK
  - Split Core & Task SDK
  - Articulate distribution strategy for Task SDK
- AIP-68 Extended Plugin Interface for React Views (Working POC)
- AIP-38 Modern Web Application
  - Updated design and infrastructure
- AIP-84 Internal API for UI Operations (some endpoints migrated)
Unknowns
- - AIP-69 Edge Executor (Initial Name: Remote Executor)
  - AIP-82 External event driven scheduling in Airflow
Won't be started

Week of 18 Nov 2024

Completion
- AIP-75 New Asset-Centric Syntax (Backend changes)
- AIP-78 Scheduler-managed backfill (Backend changes)
- AIP-65: Improve DAG history in UI (Backend changes complete)
  - API endpoints support retrieving versioned history
  - Finalized detection of observed DAG version
- AIP-66: DAG Bundles & Parsing
  - Ability to configure DAG bundles via CLI/API
  - Use DAG bundles in parsing loop
- AIP-72 Task Execution Interface aka Task SDK
  - Split ofCore and Task SDK
  - Initial incarnation of Python execution based on split
WIP
- AIP-81 Enhanced Security in CLI via Integration of API
  - Dynamic generation of API requests
  - Central API communication mechanism
  - Missing API Endpoints
- AIP-84 Internal API for UI Operations (most endpoints migrated)
- AIP-72 Task Execution Interface aka Task SDK
  - Logging support for Tasks
  - Finalize complex cases such as Dynamic Tasks for task execution support
- AIP-75 New Asset-Centric Syntax (Frontend changes)
- AIP-78 Scheduler-managed backfill (Frontend changes)
- AIP-76 Asset Partitions

Unknowns
- ...
Won't be started
- AIP-80 Explicit Template Fields in Operator Arguments

Week of 16 Dec 2024

Completion
- AIP-84 Internal API for UI Operations
  - API fully migrated to FastAPI
  - Functional and improved compared to legacy
- AIP-66: DAG Bundles & Parsing
  - Complete support for DAG bundles in task execution
  - Add support for manifest
  - Add support for some parsing control in manifest
- AIP-72 Task Execution Interface aka Task SDK
  - Complete Task Execution in python based on split Core / Task SDK
WIP
- AIP-72 Task Execution Interface aka Task SDK
  - Integration with AIP-66 for DAG fetching

Space shortcuts

Page tree

Versions Compared

Old Version 1

New Version 2

Key