Versions Compared

Old Version 11

changes.mady.by.user Bolke de Bruin

Saved on

compared with

New Version Current

changes.mady.by.user Tzu-ping Chung

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Mark as in-progress and note experiemental status

(renamed from Airflow FS)

Status

StateDraftIn Progress
Discussion Thread
Vote Thread
https://lists.apache.org/thread/wokt58k15g81cjnsytq9k1ofvspb4d5c
Created

2023-10-07

Progress Tracking

https://github.com/apache/airflow/pull/34729

Updated 2023-11-16: The AIP has been accepted as experiemental in 2.8.

Summary

With Airflow Store (AS) we want to address the challenges of fragmented code, improve XCom, and DAG processing complexity by providing a unified abstraction for accessing object storage systems. By doing so, we aim to enhance the overall user experience, promote code reusability, and simplify the development of Airflow workflows that interact with different storage backends.

...

One of the primary considerations for the adoption of AS is its integration into the core Airflow package, hosted on airflow.io. This ensures availability across DAG Processing, XCOM, Authoring and Providers. To achieve this, AS will be implemented based on the well-established fsspec library, a BSD licensed file system specification library. AFS aims to provide a standard and unified approach to working with various object storage systems through the concept of mounts and mountpoints. This unified approach will eliminate the need for users to grapple with cross-filesystem issues, such as copying data between different cloud storage providers and removing the need for specialised Operators that deal with this.

...