Page History

Status

This AIP is part of AIP-38, which aims to modernize the Airflow UI.

Motivation

Flask App BuilderAppBuilder, or FAB, has been a useful tool in developing the auth , webserver, and UI of Airflow 1 and 2, but it has started to become more burdensome than helpful.  

The FAB templates were useful to build out the first UI application, but have gotten in the way of implementing AIP-38 UI Modernization and developing new features and user experiences. Our React pages are each independent apps that are mounted inside of a FAB template html page. As of Airflow 2.9, we have to mount three separate React apps which have to be remounted on every page load because FAB controls the overall webpages. This hurts performance and prevents us from easily handling state across views. Supporting legacy FAB pages and their custom javascript also affects developer experience by forcing us to keep a complex webpack configuration for two very different javascript patterns. This hurts our ability to keep a modern standards like robust type-checking, linting, auto-formatting and hot-reloads.

Our webserver is also a mess and will be fixed in AIP-84 UI REST API

We rely on FAB as our base auth manager, but as of AIP-56 Extensible User Management auth no longer has to be part of core Airflow. That AIP also set the groundwork to swap the FAB auth manager to a KeyCloak auth manager.

The maintainers of FAB call it a "simple and rapid application development framework". Put most simply, our needs have grown beyond it's scope. Therefore, when Airflow 3.0 is released we should remove and replace Flask App Builder.

Proposal

We should take the advantage of the ability to make breaking changes in Airflow 3 and remove FAB.

Full React App

The UI becomes a single independent client-side rendered React application. It will mount once and be fast and easy to navigate around between pages. The React app can handle all of its own asynchronous requests and router management. The UI project can have a modern build process with all the industry-standard developer guide rails. We can remove outdated dependencies like momentjs and bootstrap. This is necessary to truly mark AIP-38 as done.

New Auth Manager

The default FAB Auth Manager could be replaced with our own simple auth manager for out-of-the-box development or with a KeyCloak Auth Manager like discussed in AIP-56. KeyCloak brings much more functionality than FAB too Perhaps the FAB Auth Manager can be kept as a provider plugin. That still needs to be explored.

Many plugins, especially UI-facing ones, use FAB to generate views or create custom models. Those will have to be rewritten to interface with a modern React UI and a new webserver REST API. AIP-68 New Plugin Interface goes into more detail on how this can be implemented and provide new ways for plugins to connect into the Airflow UI.

Plugins

This AIP should be implemented jointly with AIP-68 Extended Plugin Interface for React Views. Removing FAB will mean we will lose the appbuilder_views, and appbuilder_menu_items attributes on a Plugin. AIP-68 will need to be extended here to replace the lost FAB functionality with new javascript powered views. Out-of-the-box, old plugins will not be able to add views to the UI. We will try to get a FAB provider which can render legacy UI plugins and then send the views to the UI to render in an iframe.

How are users affected?

We’ve relied on FAB for authentication (authn) and authorization (authz), but as of AIP-56 Extensible User Management FAB no longer has to be part of core Airflow, as we can rely on other systems for those functions.

However, today FAB does provide the most commonly used authn/authz for Airflow, and there is certainly value in maintaining backwards compatibility for that purpose and for custom plugins. So let’s set the stage for fully removing FAB later, while maintaining a smooth upgrade path for users today.

Goals

• Have a smooth migration path from Airflow 2 to Airflow 3, in particular for authn/authz
• Set ourselves up for a true “FAB-free” future (while also making it possible today)
• If possible, maintain backwards compatibility for plugins

Proposal

The UI components we use from FAB (e.g. CRUD for TIs) will be handled by AIP-38. The rest of this proposal will address the remaining uses.

Authentication / Authorization

The default FAB auth manager and its adjacent entities will be fully moved into the FAB provider. This is mostly done already, but parts of core still expect FAB, so we can do a fresh pass on this (e.g. remove the need for the ApplessAirflowSecurityManager).

This means an Airflow instance could be fully functional without FAB even being installed in the environment, and instead rely on a custom auth manager or other community manager auth manager.

In order to power local environments, which typically do not require authn or authz, core Airflow will contain a minimal auth backend that does not require any authentication and allows the user to perform all actions.

The FAB auth manager can be installed and used, if desired, for more complex environments.

Views from plugins

While AIP-68 will add support for Airflow 3 native plugins, backwards compatibility for plugins using FAB views (`appbuilder_views`) will be maintained, if possible.

Ideally, they continue working as-is. However, we reserve the right to remove support completely if we hit insurmountable hurdles during implementation - this is a best effort goal ultimately.

Menu items from plugins

A new interface for menu items will be added to plugins, as the current interface (`appbuilder_menu_items`) is specific to FAB. The FAB specific menu items interface will be deprecated in favor of a React menu item plugin.

Menu items from FAB (either from views in plugins or default menu items like the security tab) will be available in the Airflow 3 UI. We will likely have an API endpoint to expose the FAB menu items, which will allow the react UI to render them appropriately. This maintains backcompat for the plugins, and also allows the security pages to continue functioning.

How?

If the FAB provider is installed and the FAB auth manager is configured or plugins with FAB views are present, during webserver initialization the provider will create an appbuilder app to be used for that purpose.  We will do our best to avoid extra work if possible, but we will iron out the specifics during implementation. For example, the loading order of plugins and providers may make it difficult to detect if a plugin defines FAB views when the FAB provider is being loaded.

Then the FAB auth manager is used, there are some db tables that need to be created or migrated. Instead of being tightly integrated into the sqlalchemy models in core, the FAB provider will take complete ownership of its db tables and maintain its own db cli commands (like migrate/reset/etc).

Why?

You may be wondering why this is worth it - can’t we instead just remove FAB altogether? While we could, today nearly all auth uses FAB. And we do not have a well defined replacement (something like Keycloak) scoped out, even if we do have a good interface to do it technically (auth manager). This means that, practically, to get adoption of Airflow 3 we will need backwards compatibility with the existing auth configuration in Airflow 2. A migration plan for migrating from the FAB auth manager to something else can be developed and made available to the Airflow community at a later date, and we can also end support for the FAB provider once we are comfortable that we have a solid replacement and migration path in place. Compatibility for plugins is just a bonus to make the Airflow 3 migration easier - the authn/authz aspect is the main driver for keeping FAB around.

How are users affected?

• Less complex dependencies - FAB is no longer required for core
• Minimal migration pain for users from Airflow 2 to 3
• Using the FAB provider requires more deployment steps (e.g. separate db commands)
• Users will lose the ability to use the FAB Auth Manager and any legacy UI plugins out-of-the-box
• Users will get a fast and responsive UI
• Contributors will have an easy time making changes to Airflow with more standardized linting and testing
• A lot of views are subject to be redesigned as we rebuild them
• The UI completely relying on rest APIs vs FAB will result in the public REST API becoming more expansive and powerful for 3rd party users

What defines this AIP as done?

• Flask AppBuilder is no longer needed required to run the webserver and UI
• The python webserver backend is separated from the typescript React UI
• New UI plugins are feature-complete with FAB plugins (ie: can add new top-nav buttons and menus with full page views)
• existing FAB auth manager is fully functional from the FAB provider, but not required by core Airflow
• If possible, FAB plugin compatibility is maintainedA new default auth manager is feature-complete with the FAB auth features we use