DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
Nothing in this page should be taken as a commitment of what will be in Airflow 2.0.0, instead this should be read as the Airflow project's goal for what will make it to that release. This is also still a work-in-progress so will change over time.
High-level Features
The following is the list of features that we are targeting for Airflow 2.0:
- Knative Executor
- Improve Webserver performance
- Enhanced real-time UI
- Improve Scheduler performance
- Extend/finish the API
- Production Docker image
Compelted AIPs
The following AIPs have been completed and will be in Airflow 2.0
| Title | Title | State | Discussion Thread | JIRA | InRelease | Created |
|---|---|---|---|---|---|---|
| AIP-26 Production-ready Airflow Docker Image | Complete | AIP-26 Discussion | ||||
| AIP-9 Automated Dependency Management | Completed | AIP-9 Automated Dependency Management | $action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate()) | |||
| AIP-21: Changes in import paths | Completed | $action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate()) | ||||
| AIP-34 TaskGroup: A UI task grouping concept as an alternative to SubDagOperator | Completed | $action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate()) | ||||
| AIP-15 Support Multiple-Schedulers for HA & Better Scheduling Performance | Completed | Multiple Schedulers - "scheduler_lock" A Naive Multi-Scheduler Architecture Experiment of Airflow [PROPOSAL][AIP-15 Support Multiple-Schedulers for HA & Better Scheduling Performance] | $action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate()) | |||
| AIP-8 Split Providers into Separate Packages for Airflow 2.0 | Completed | $action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate()) | ||||
| AIP-32: Airflow REST API | Completed | $action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate()) | ||||
| AIP-17: Consolidate and de-duplicate sensor tasks in airflow Smart Sensor | Completed | [Discuss] Airflow sensor optimization | $action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate()) | |||
| AIP-16: Use nested commands in CLI | Completed | $action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate()) | ||||
| AIP-3 Drop support for Python 2 | Completed | [DISCUSS] AIP-3: Drop Python2 support in Airflow 2.0 | $action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate()) | |||
| AIP-10 Multi-layered and multi-stage official Airflow CI image | Completed | Mutli-layered official image for Airflow | $action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate()) |
Planned AIPs
The following AIPs have been voted on by the Airflow community and are planned to be included in Airflow 2.0
| Title |
|---|
| No content found. |
Candidate AIPs
The following AIPs are under consideration for inclusion in Airflow 2.0
| Title |
|---|
| No content found. |
Needs more detail/AIP
This is a list of new features that we would like to see in Airflow 2.0 (that don't yet have AIPs or Jira tickets). These will need AIPs creating and voting on before they make it in to the roadmap.
In no particular order:
- Rework Subdags to be less "bolted-on" and more native to the scheduler
There are all sorts of edge cases around subdags that result from the tasks in the subdag being run by another executor, instead of being handled and scheduled by the core Scheduler. We should make the scheduler "see in" to the Subdags and make it responsible for scheduling tasks. This should make subdags less error prone and more predictable. It may involve replacing/significantly changing the SubDagOperator - Move (tested) components out of contrib folder
https://lists.apache.org/thread.html/c880ef89f8cb4a0240c404f9372615b998c4a4eeca342651927d596c@%3Cdev.airflow.apache.org%3E - Filter passwords/sensitive info from logs.
Jenkins does this if the password comes from a connection - it would be good if we could do this too - Allow Backfill runs to be handled by the scheduler/triggered from UI
It would be nice to not need console access to runairflow backfill, and to have not it not stop if the SSH session is closed.
Lots of details to work out here though around how this would work, where would it show up in UI, priority of tasks, ways of reducing concurrency/load to allow normal tasks to run etc. - Rationalize HA in Connections
Right now it is possible to create multiple connections with the same ID and some Connections/hooks will support this and pick a random one from the list. This feature isn't well documented or understood (and the CLI doesn't support it as well as the UI for instance) so we should examine if this makes sense, or if we should support it individually in certain connection types instead. - Publish "official" production ready docker image
Many people deploy Airflow via docker and currently use puckel/docker-airflow, but it would be nice if we published our own one (mostly just so that we can be more prompt about publishing new releases to docker hub too) - Make setting up HTTPS connections easier/more expected
AIRFLOW-2910 - Getting issue details... STATUS - Front end/"browser" testing
The Airflow UI is non trivial and there have been a number of JS/html bugs that could have been caught by better front-end testing.
It has been suggested to look at Cypress for this over Selenium. What ever we choose we need to pay careful attention to avoid slow or flakey UI tests.