Nothing in this page should be taken as a commitment of what will be in Airflow 2.0.0, instead this should be read as the Airflow project's goal for what will make it to that release. This is also still a work-in-progress so will change over time.

High-level Features

The following is the list of features that we are targeting for Airflow 2.0:

  1. Knative Executor (handled by KEDA)
  2. Improve Webserver performance
  3. Enhanced real-time UI
  4. Improve Scheduler performance and reliability
  5. Extend/finish the API
  6. Production Docker image

Completed AIPs

The following AIPs have been completed and will be in Airflow 2.0

TitleTitleStateDiscussion ThreadJIRAInReleaseCreated
AIP-26 Production-ready Airflow Docker Image CompleteAIP-26 Discussion

AIRFLOW-5606 - Jira project doesn't exist or you don't have permission to view it.

  
AIP-9 Automated Dependency Management CompletedAIP-9 Automated Dependency Management
 

$action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate())

AIP-21: Changes in import paths Completed
[VOTE] Changes in import paths

AIRFLOW-4733 - Jira project doesn't exist or you don't have permission to view it.

 

$action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate())

AIP-34 TaskGroup: A UI task grouping concept as an alternative to SubDagOperator Completed
Pony Mail
  

$action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate())

AIP-15 Support Multiple-Schedulers for HA & Better Scheduling Performance Completed

Multiple Schedulers - "scheduler_lock"

A Naive Multi-Scheduler Architecture Experiment of Airflow

[PROPOSAL][AIP-15 Support Multiple-Schedulers for HA & Better Scheduling Performance]

  

$action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate())

AIP-8 Split Providers into Separate Packages for Airflow 2.0 Completed

AIP-8 Split Hooks/Operators into Separate Packages

Voting discussion.

AIRFLOW-3644 - Jira project doesn't exist or you don't have permission to view it.

 

$action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate())

AIP-32: Airflow REST API Completed
[PROPOSAL][AIP-32] Airflow REST API

https://github.com/apache/airflow/issues/8107

 

$action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate())

AIP-17: Consolidate and de-duplicate sensor tasks in airflow Smart Sensor Completed[Discuss] Airflow sensor optimization

AIRFLOW-3964 - Jira project doesn't exist or you don't have permission to view it.

 

$action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate())

AIP-16: Use nested commands in CLI Completed

[VOTE] AIP-16: CLI: Use nested commands instead of flags

CLI: Use nested commands instead of flags

AIRFLOW-3998 - Jira project doesn't exist or you don't have permission to view it.

 

$action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate())

AIP-3 Drop support for Python 2 Completed[DISCUSS] AIP-3: Drop Python2 support in Airflow 2.0

AIRFLOW-4196 - Jira project doesn't exist or you don't have permission to view it.

 

$action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate())

AIP-10 Multi-layered and multi-stage official Airflow CI image CompletedMutli-layered official image for Airflow

AIRFLOW-3718 - Jira project doesn't exist or you don't have permission to view it.

 

$action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate())

Planned AIPs

The following AIPs have been voted on by the Airflow community and are planned to be included in Airflow 2.0

Title
No content found.

Candidate AIPs

The following AIPs are under consideration for inclusion in Airflow 2.0

Title
No content found.


Needs more detail/AIP

This is a list of new features that we would like to see in Airflow 2.0 (that don't yet have AIPs or Jira tickets). These will need AIPs creating and voting on before they make it in to the roadmap.

In no particular order:

  • Rework Subdags to be less "bolted-on" and more native to the scheduler

    There are all sorts of edge cases around subdags that result from the tasks in the subdag being run by another executor, instead of being handled and scheduled by the core Scheduler. We should make the scheduler "see in" to the Subdags and make it responsible for scheduling tasks. This should make subdags less error prone and more predictable. It may involve replacing/significantly changing the SubDagOperator

  • Move (tested) components out of contrib folder
    https://lists.apache.org/thread.html/c880ef89f8cb4a0240c404f9372615b998c4a4eeca342651927d596c@%3Cdev.airflow.apache.org%3E

  • Filter passwords/sensitive info from logs.

    Jenkins does this if the password comes from a connection - it would be good if we could do this too
  • Allow Backfill runs to be handled by the scheduler/triggered from UI

    It would be nice to not need console access to run airflow backfill, and to have not it not stop if the SSH session is closed.

    Lots of details to work out here though around how this would work, where would it show up in UI, priority of tasks, ways of reducing concurrency/load to allow normal tasks to run etc.
  • Rationalize HA in Connections

    Right now it is possible to create multiple connections with the same ID and some  Connections/hooks will support this and pick a random one from the list. This feature isn't well documented or understood (and the CLI doesn't support it as well as the UI for instance) so we should examine if this makes sense, or if we should support it individually in certain connection types instead.
  • Publish "official" production ready docker image

    Many people deploy Airflow via docker and currently use puckel/docker-airflow, but it would be nice if we published our own one (mostly just so that we can be more prompt about publishing new releases to docker hub too)
  • Make setting up HTTPS connections easier/more expected

    AIRFLOW-2910 - Getting issue details... STATUS
  • Front end/"browser" testing

    The Airflow UI is non trivial and there have been a number of JS/html bugs that could have been caught by better front-end testing.

    It has been suggested to look at Cypress for this over Selenium. What ever we choose we need to pay careful attention to avoid slow or flakey UI tests.
  • No labels

3 Comments

  1. Jarek Potiuk

    May 04, 2019

    I would add few things here for consideration. Maybe we should not add then all to 2.0 to make sure 2.0 is released in a timely fashion, but I see those as important ones that add a lot of value:

    • AIP-7 Simplified development workflow - which is a follow-up after AIP-10 Multi-layered and multi-stage official Airflow CI image - which might help with faster on-boarding of contributors.
    • Security Process in general - I think we have not paid a lot of attention to security as a community but we should improve not only tooling, like detection of security issues, signing and releasing releases etc. but also process of dealing with security issues. How the detected security issues are communicated internally, how we involve our users with security vulnerabilities - responsive disclosure etc. how we integrate it all with CVE process. I don't think it's clear how this is done even now and maybe we should check how other Apache projects are doing that and learn and apply best practices from them
    • Some way of addressing consistency between DAGs for scheduler (i.e. scanning the folder and making sure related DAGs are consistent). This is some cross between AIP-20 DAG manifest, AIP-15 Support Multiple-Schedulers for HA & Better Scheduling Performance (already mentioned in the doc above) and Deprecated- AIP-5 DagFetcher.
    • Some form of AIP-12 Persist DAG into DB - not sure exactly what form, but it would be great if the DAGs can be somehow "packaged" and provided to the worker rather than synced as files in DAG folder. 

  2. Kamil Bregula

    Apr 14, 2020

    Latest discussion take place on the mailing list: https://lists.apache.org/thread.html/r0abba3669962f101d787ad793611ba436d35c8e022aa565705778b7d%40%3Cdev.airflow.apache.org%3E

  3. Tony Brookes

    Jul 08, 2020

    I would see it as critically important that we are able to view dag runs "as they ran" not "as what they would look like based on evaluating the DAG now."  Whilst it's excellent that the DAG is now persisted to the DB, it is, sadly, persisted using the DAG as the primary key, NOT the DAG and the Dag RUN.

    This makes it impossible for me to generate "only the tasks I need for this run of the DAG" which would help me greatly.  It also means that I can't get an accurate picture of prior DAG runs because tasks that are NOT generated "now" look like they never happened at all.  Similarly tasks which are generated now but weren't in the actual run look like they didn't run and create confusion about why downstream tasks ever started if their dependencies never ran.

    I think it's really important that we get a grip on this and I'd be happy to start contributing as a way of making it happen (if someone could assist me in getting going.)  I rely on Airflow a lot and am more than happy to give back. (smile)