Airflow Improvements Proposals

This page describes a proposed Airflow Improvement Proposal (AIP) process for proposing a major change to Airflow.

To create your own AIP, click on Create AIP. If you don't have permission, please send an email with your Wiki ID to dev@airflow.apache.org and request permission

Completed AIPs

Title	Discussion Thread	JIRA	Created
AIP-3 Drop support for Python 2	[DISCUSS] AIP-3: Drop Python2 support in Airflow 2.0	Unable to render Jira issues macro, execution error.	$action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate())
AIP-6 Enforce the usage of a code formatter & linter	AIP-6: Enforce the usage of a code formatter		$action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate())
AIP-7 Simplified development workflow	AIP-7 Simplified development workflow	Unable to render Jira issues macro, execution error.	$action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate())
AIP-8 Split Providers into Separate Packages for Airflow 2.0	AIP-8 Split Hooks/Operators into Separate Packages Voting discussion.	Unable to render Jira issues macro, execution error.	$action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate())
AIP-9 Automated Dependency Management	AIP-9 Automated Dependency Management		$action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate())
AIP-10 Multi-layered and multi-stage official Airflow CI image	Mutli-layered official image for Airflow	Unable to render Jira issues macro, execution error.	$action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate())
AIP-11 Create a Landing Page for Apache Airflow	[PROPOSAL] Add a landing page for Apache Airflow	Unable to render Jira issues macro, execution error.	$action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate())
AIP-15 Support Multiple-Schedulers for HA & Better Scheduling Performance	Multiple Schedulers - "scheduler_lock" A Naive Multi-Scheduler Architecture Experiment of Airflow [PROPOSAL][AIP-15 Support Multiple-Schedulers for HA & Better Scheduling Performance]		$action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate())
AIP-16: Use nested commands in CLI	[VOTE] AIP-16: CLI: Use nested commands instead of flags CLI: Use nested commands instead of flags	Unable to render Jira issues macro, execution error.	$action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate())
AIP-17: Consolidate and de-duplicate sensor tasks in airflow Smart Sensor	[Discuss] Airflow sensor optimization	Unable to render Jira issues macro, execution error.	$action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate())
AIP-21: Changes in import paths	[VOTE] Changes in import paths	Unable to render Jira issues macro, execution error.	$action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate())
AIP-23 Migrate out of Travis CI	Discussion Thread	Unable to render Jira issues macro, execution error.	$action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate())
AIP-24 DAG Persistence in DB using JSON for Airflow Webserver and (optional) Scheduler	Airflow DAG Serialisation	Unable to render Jira issues macro, execution error.	$action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate())
AIP-27 Migrate to pytest	[PROPOSAL] Migrate to Pytest	Unable to render Jira issues macro, execution error.	$action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate())
AIP-31: "TaskFlow API" for clearer/simpler DAG definition	[DISCUSS] Airflow functional DAGs Slack thread		2020-02-21
AIP-32: Airflow REST API	[PROPOSAL][AIP-32] Airflow REST API	https://github.com/apache/airflow/issues/8107	$action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate())
AIP-33 Secrets backend	[DISCUSS] AIP-33 secrets backend	Unable to render Jira issues macro, execution error.	$action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate())
AIP-34 TaskGroup: A UI task grouping concept as an alternative to SubDagOperator	Pony Mail		$action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate())
AIP-39 Richer scheduler_interval	Scoping out a new feature for 2.1: improving schedule_interval [DISCUSS][AIP-39] Richer (and pluggable) schedule_interval on DAGs		$action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate())
AIP-40: Deferrable ("Async") Operators	[DISCUSS][AIP-40] Deferrable ("Async") Operators		$action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate())
AIP-42 Dynamic Task Mapping	https://lists.apache.org/thread/xhq5o3b3h879cwsc93xwq0836mwc5k6p		$action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate())
AIP-43 DAG Processor separation	https://lists.apache.org/thread/rgovdphjrxtcqfmcfdrbfntwrdr2nj71		$action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate())
AIP-43 DAG Processor separation	https://lists.apache.org/thread/rgovdphjrxtcqfmcfdrbfntwrdr2nj71		$action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate())
AIP-48 Data Dependency Management and Data Driven Scheduling	https://lists.apache.org/thread/vnjv32zv3fqqp92t5nqf0qqy03xphwm8		$action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate())
AIP-53 OpenLineage in Airflow	https://lists.apache.org/thread/2brvl4ynkxcff86zlokkb47wb5gx8hw7		$action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate())

Accepted AIPs

AIPs which have been voted on and accepted for inclusion in Airflow, but not yet (fully) implemented.

Title	Discussion Thread	JIRA	Created
AIP-26 Production-ready Airflow Docker Image	AIP-26 Discussion	Unable to render Jira issues macro, execution error.
AIP-38 Modern Web Application	[DISCUSS][AIP-38 Modern Web Application]		$action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate())
AIP-44 Airflow Internal API	https://lists.apache.org/thread/nsmo339m618kjzsdkwq83z8omrt08zh3		$action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate())
AIP-47 New design of Airflow System Tests	https://lists.apache.org/thread/htd4013yn483qfhwv11vc26jpf2yvjph		$action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate())
AIP-49 OpenTelemetry Support for Apache Airflow	[DISCUSSION] AIP-49 OpenTelemetry Support for Apache Airflow		$action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate())
AIP-50 Trigger DAG UI Extension with Flexible User Form Concept	https://lists.apache.org/thread/kxkctcbh9drfw065dgvr673zl0xyfl3r		$action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate())
AIP-51 Removing Executor Coupling from Core Airflow	https://lists.apache.org/thread/yd4tcgyqft664dt8clgx48gyvfzddnjh https://github.com/apache/airflow/discussions/27241		$action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate())

AIPs under discussion

Title	Discussion Thread	JIRA	Created
[WIP] AIP-55 Rule-based timetable with logical composition
AIP-5 Remote DAG Fetcher	https://lists.apache.org/thread.html/224d1e7d1b11e0b8314075f21b1b81708749f2899f4cce5af295e8a8@%3Cdev.airflow.apache.org%3E	Unable to render Jira issues macro, execution error.
AIP-14 Create composable operators using common interfaces		Unable to render Jira issues macro, execution error.	$action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate())
AIP-20 DAG manifest		Unable to render Jira issues macro, execution error.	$action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate())
AIP-22: Group ORM models by their logical usage instead of type	https://lists.apache.org/thread.html/fc21e13413a4a11b2c490b0e2f59dd83239711433599fe8693b98d02@%3Cdev.airflow.apache.org%3E		$action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate())
AIP-35 Add Signal Based Scheduling To Airflow	[DISCUSS][AIP-35] Add Signal Based Scheduling To Airflow		$action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate())
AIP-36 DAG Versioning			$action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate())
AIP-45 Remove double dag parsing in airflow run	https://lists.apache.org/thread/tc37cgrx2tojv3zgzokz06f5ypk0y2hg		$action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate())
AIP-46 Runtime isolation for airflow tasks and dag parsing	https://lists.apache.org/thread/9kw13cxcg06p37shv57hsomnx6zognoc		$action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate())
AIP-54 Multi tenancy - View Level Access Control
AIP-56 Extensible user management	https://lists.apache.org/thread/ck8dsj5w82lvr0cpwr4wlptmydqwnsqc
AIP-57 Refactor SLA Feature	https://lists.apache.org/thread/kn1rb9l70jk16j83fycp68l2g70wnhzl https://lists.apache.org/thread/kj33fwd9otbbm52my9j3y39rp547g0n5 https://lists.apache.org/thread/l2ho7w18d11vcnon7dxoxghzxhtjw5hf (Multiple threads were created on the topic)
AIP-59 Performance tests framework
AIP-60 Standard URI representation for Airflow Datasets	https://lists.apache.org/thread/rf6c80ljjkml0l15h2jys7k713q3os1d
AIP-61 Hybrid Execution	https://lists.apache.org/thread/94sg7l4m3qjk4b3vfq3lr94oc5fs9q4j

Abandoned AIPs

Title	State	Discussion Thread	JIRA	Created
AIP-2 Simplify process launching	Draft			$action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate())
AIP-12 Persist DAG into DB	Abandoned in favour of AIP-24	[DISCUSS] AIP-12 Persist DAG into DB	Unable to render Jira issues macro, execution error.	$action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate())
AIP-13: OpenAPI 3 based API definition	Abandoned in favour of AIP-32		Unable to render Jira issues macro, execution error.	$action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate())
AIP-18 Persist all information from DAG file in DB	Abandoned in favour of AIP-24	TBD	TBD	$action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate())
AIP-19 Making the webserver stateless	Abandoned in favour of AIP-24			$action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate())
AIP-25 The Knative Executor	Abandoned			$action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate())
AIP-28: Add AsyncExecutor option	Abandoned in favour of AIP-40: Deferrable ("Async") Operators	apache-airflow.slack.com - sig-async-operators	Unable to render Jira issues macro, execution error.	$action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate())
AIP-29: AWS Fargate Executor	Abandoned in favour of AIP-51		Unable to render Jira issues macro, execution error.
AIP-37 Virtualenv management inside Airflow	Draft			$action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate())
Deprecated- AIP-5 DagFetcher	Abandonded in favour of new https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-5+Remote+DAG+Fetcher		Unable to render Jira issues macro, execution error.	$action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate())

(The above list is automatically generated based on page labels. Do not edit manually)

Space shortcuts

Page tree

Completed AIPs

Accepted AIPs

AIPs under discussion

Abandoned AIPs