AIP-68 Extended Plugin Interface for Custom Grid View Panels

Status

Motivation

As we have already contributed to UI in various PRs and still closely monitor and (attempt) to support the completion of AIP-38 Modern Web Application we are looking forward to close gaps in UI for a better business integration. From a perspective of business users the UI might be very technical. You can see task details, logs and run-times. This is very good for a pipeline developer and operations teams, business users would also like to see custom details of the workflow that is represented behind the scene.

Today this is mostly compensated by embedding Airflow behind other UI's or business applications and just drive or schedule a pipeline, hiding the UI to business users. But Airflow UI - especially after AIP-38 - does not need to be shy and could be opened up also for (technical oriented) business users.

As Airflow UI is generic and allows deep customization's of Operators, Provider Packages, Workflows, the UI is quite generic. It would be great to be able to extend the Grid view to populate custom information via adding widgets or details besides adding full web pages or the need to populate another UI in front of the workflow.

Considerations

As we had implemented UI Plugins via Flask Appbuilder pages, integrating with the menu and web page in the past, also as having contributed AIP-50 Trigger DAG UI Extension with Flexible User Form Concept in the past - allowing to provide overview and customized details to business users can be made with the following options:

1. Implement a "Business Targeted" UI in front of Airflow - this might be exactly tailored to the UI and needs a user has to see what is going on with his business processes and details. Such a UX might offer the possibility to trigger workflows manually ( Here the trigger UI would need to be re-implemented mostly, triggering via API) and also as kind of reporting a business user could see his status of success ( For details a couple of state information points must be sourced from Airflow via API, parts of the workflow must be re-modeled in a custom UI and for details probably a deep link to Airflow UI is needed, not to 50-80% re-implement UI features). Sometimes Reporting tools like Grafana, Power BI etc. might serve the need to aggregate the results from resulting systems or also provide dashboards of workflows from Airflow database. Nevertheless a lot of UI, authentication might need to be implemented for the customer use case with overlap to Airflow - which means a big investment.
2. If Airflow delivers most business workflow information and just an overview is missing, you could assume that 80% of UI is reasonable but 20% is missing. In such case some Flask UI Plugins could be added to render custom web pages which allow an overview for the business user, maybe alongside a Grafana, Power BI etc. reporting for statistics. But most detailed information behind an overview might be visible in Grid view. But the Grid view provides mostly textual data on the technical level based on logs, properties and statistics. Sometimes you could use Operator Extra Links to connect the basic technical information with an external system showing more details.
3. Extensions as proposed in the AIP below, allowing the option to extend the Grid view for business information based on use case needs via an extended Plugin interface

What change do you propose to make?

I propose that the existing Airflow Plugin Interface is extended for the purpose of adding custom panels based on business needs into Grid views. Compared to separate Flask pages, business specific information could be added at the right place into panels where needed.

In detail the following proposals for plugin extensions are made:

• Custom Home Page (Separating Click on Airflow Logo from "DAGs View" - today both links show the DAG overview - there is no option ot have a custom landing page - Similar like "Cluster Activity" allow a plugin to render a user overview page e.g. "Open Activities / Business Problems / My Workflows running/completed"
• DAG Overview
  • Custom Panel(s) on DAG Details
• DAG Run
  • Custom Panel(s) on DAG Run Details
  • Custom Tab (allowing to fill the whole side) on DAG Run
• Task Instance
  • Custom Panel(s) on Task Instance Details
  • Custom Tab (allowing to fill the whole side) on Task Instance
    
• Trigger DAG Run
  • Custom HTML rendered field entry (Which was removed from initial AIP-50 implementation because of security)
  • Note: This assumes that the existing Trigger DAG Run page is converted from previous HTML rendering to ReactJS

Examples - Extended "Donkey views":

DAG Run View	Task Instance View
Custom Panel Area populated for DAG Run	Custom Panel and Tab for Donkey Statistics and History in Task Instance View
Custom Tab Added to DAG Run

Technical specifics assumed to provide the panels:

• Plugin provides API endpoints to load HTML fragments to be inserted into panels (lazy loaded)
• Due to the integration as Plugin the content could be generated with access to all DAG/Task Instance context information or also query external systems
• Security and deployment is in the responsibility and ownership of Deployment Manager
• Panel details and selection is made on DAG name/tags and Operator type

What problem does it solve?

The new/current Grid view is generic and shows all technical information of an Operator - with the option to extend via Plugins the UI can be enriched with business relevant information above the current Operator Extra Links

Why is it needed?

Save users to develop another UI on top if Airflow serves for 60% of needs and UX is required to be added to close the gap

Are there any downsides to this change?

• (Small) Added complexity
• Risks if bad plugins are added by developers
  • Risk of performance problems (e.g. complex back-end logic slowing down web server)
  • Security problems generated by bad implemented plugins
  • XSS problems by bad HTML code generated by Plugins

Which users are affected by the change?

None. Plugin would be optional to be extended

How are users affected by the change? (e.g. DB upgrade required?)

Assumed no DB changes. All by Python code, templating and API extensions

Other considerations?

An example should be provided with the extended Plugin interface incluing extended documentation, Also adding some considerations to cover/compensate the risks (e.g. authentication, XSS best practices)

What defines this AIP as "done"?

• The extended Plugin interface is with the described features available
• Users can implement a plugin.
• Documentation is complete