AIP-50 Trigger DAG UI Extension w/ Flexible User Form Concept

Status

Motivation

As user of Airflow for our custom workflows we often use DagRun.conf attributes to control content and flow. Current UI allows (only) to launch via REST API with given parameters or using a JSON structure in the UI to trigger with parameters. This is technically feasible but not user friendly. A user needs to model, check and understand the JSON and enter parameters manually without the option to validate before trigger.

Similar like Jenkins or Github/Azure pipelines we desire an UI option to trigger with a UI and specifying parameters. We'd like to have a similar capability in Airflow.

Considerations

This AIP was created after working a longer time with an UI plugin which we implemented customized allowing users to "easily" trigger DAGs with parameters. More and more we realize that every use case and demo we make requires an simple entry form to show "how easy" it is (not to scare users and have them running away). We propose to "productize" the feature from a plugin and merge it into the Airflow Core UI - hoping that more people can use this feature, having a lower entry barrier for "new" users and also to close gaps from standard UI path to trigger a DAG today compared to how the UI plugin is integrated.

We also had a separate UI just to collect parameters and trigger the Airflow DAG in the backend via REST API - but this was a full sepatration of the user from Airflow and at the end requires a very redundant UI being developed and maintained.

What change do you propose to make?

Implementation Part 1) Specifying Trigger UI on DAG Level

We propose to extend the DAG class with an additional optional attribute so that UI(s) (one or multiple per DAG) can be specified in the DAG.

• Attribute name proposal: trigger_ui
• Type proposal: Union[TriggerUIBase, List[TriggerUIBase] (One or a list of UI definition inherited from an abstract UI class which implements the trigger UI)
• Default value proposal: [TriggerNoUI(), TriggerJsonUI()] (Means the current/today's state, user can pick to trigger with or without parameters)

With this extension the current behavior is continued and users can specify if a specific or multiple UIs are offered for the Trigger DAG option.

Implementation Part 2) UI Changes for Trigger Button

The function of the trigger DAG button in DAG overview landing ("Home" / templates/airflow/dags.html) as well as DAG detail pages (grid, graph, ... view / templates/airflow/dag.html) is adjusted so that:

1. If there is a single Trigger UI specified for the DAG, the button directly opens the form on click

2. If a list of Trigger UIs is defined for the DAG, then a list of UI's is presented, similar like today's drop-down with the today's two options (with and without parameters).

Menu names for (2) and URLs are determined by the UI class members linked to the DAG.

Implementation Part 3) Standard implementations for TriggerNoUI, TriggerJsonUI

Two implementations for triggering w/o UI and parameters and the current JSON entry form will be migrated to the new UI structure, so that users can define that one, the other or both can be used for DAGs.

Name proposals:

1. TriggerUIBase: Base class for any Trigger UI, defines the base parameters and defaults which every Trigger UI is expected to provide:

   • url_template: URL template (into which the DAG name is inserted to direct the user to)
   • name: Name of the trigger UI to display in the drop-down
   • description: Optional descriptive test to supply as hover-over/tool-tip)

2. TriggerNoUI (inherits TriggerUIBase): Skips a user confirmation and entry form and upon call runs the DAG w/o parameters (DagRun.conf = {})

3. TriggerJsonUI (inherits TriggerUIBase): Same like the current UI to enter a JSON into a text box and trigger the DAG. Any valid JSON accepted.

Implementation Part 4) Standard Implementation for Simple Forms (Actually core new feature)

Implement/Contribute a user-definable key/value entry form named TriggerFormUI (inherits TriggerUIBase) which allows the user to easily enter parameters for triggering a DAG. Form could look like:

   Parameter 1: <HTML input box for entering a value>
   (Optional Description and hints)
   
   Parameter 2: <HTML Select box of options>
   (Optional Description and hints)
   
   Parameter 3: <HTML Checkbox on/off>
   (Optional Description and hints)
   
   <Trigger DAG Button>

The resulting JSON would use the parameter keys and values and render the following DagRun.conf and trigger the DAG:

{
  "parameter_1": "user input",
  "parameter_2": "user selection",
  "parameter_3": true/false value
}

The number of form values, parameter names, parameter types, options, order and descriptions should be freely configurable in the DAG definition.

The trigger form should provide the following general parameters (at least):

• name: The name of the form to be used in pick lists and in the headline
• description: Descriptive test which is printed in hover over of menus and which will be rendered as description between headline and form start
• (Implicitly the DAG to which the form is linked to which will be triggered)

The trigger form elements (list of elements can be picked freely):

• General options of each form element (Base class TriggerFormUIElement:

  • name (str): Name of the parameter, used as technical key in the JSON, must be unique per form (e.g. "param1")
  • display (str): Label which is displayed on left side of entry field (e.g. "Parameter 1")
  • help (Optional[str]=Null): Descriptive help text which is optionally rendered below the form element, might contain HTML formatting code
  • required (Optional[bool]=False): Flag if the user is required to enter/pick a value before submission is possible
  • default (Optional[str]=Null): Default value to present when the user opens the form

• Element types provided in the base implementation

  • TriggerFormUIString (inherits TriggerFormUIElement): Provides a simple HTML string input box.
  • TriggerFormUISelect (inherits TriggerFormUIElement): Provides a HTML select box with a list of pre-defined string options. Options are provided static as array of strings.
  • TriggerFormUIArray (inherits TriggerFormUIElement): Provides a simple HTML text area allowing to enter multiple lines of text. Each line entered will be converted to a string and the strings will be used as value array.
  • TriggerFormUICheckbox (inherits TriggerFormUIElement): Provides a HTML Checkbox to select on/off, will be converted to true/false as value
  • TriggerFormHidden (inherits TriggerFormUIElement): Provides a pre-defined key/valure w/o displaying any form field, a.k.a HTML hidden form element

• Other element types (optionally, might be added later?) for making futher cool features - depending on how much energy is left

  • TriggerFormUIHelp (inherits TriggerFormUIElement): Provides no actual parameter value but allows to add a HTML block of help
  • TriggerFormUIBreak (inherits TriggerFormUIElement): Provides no actual parameter value but adds a horizontal splitter
  • Adding the options to validate string values e.g. with a RegEx
  • Allowing to provide int values (besides just strings)
  • Allowing to have an "advanced" section for more options which the user might not need in all cases
  • Allowing to view the generated DagRun.conf so that a user can copy/paste as well
  • Allowing to user extend the form elements...

Example view how today's solution is made via a UI plugin to integrate into Airflow UI (but is not linked into the standard "Trigger DAG" button):

Note that the advanced parameters and generated Job Config are hidden per default and need to be un-folded

Implementation Part 5) (Optional) Extended for Templated Form based on the Simple form but uses fields to run a template through Jinja

Implement (optionally, might be future extension as well?) a TriggerTemplateFormUI (inherits TriggerFormUI) which adds a Jinja2 JSON template which will be templated with the collected form fields so that more complex DagRun.conf parameter structures can be created on top of just key/value

Implementation Part 6) Examples

Provide 1-2 example DAGs which show how the trigger forms can be used. Adjust existing examples as needed.

Implementation Part 7) Documentation

Provide needed documentation to describe the feature and options. This would include an description how to add custom forms above the standards via Airflow Plugins and custom Python code.

What problem does it solve?

With this proposal the functional gap is closed allowing to add user friendly trigger UI forms for staring DAGs with parameters (w/o the need that every user must understand the specific required JSON structure to trigger a DAG). Also it adds missing functionality which users know and expect to have from e.g. CI/CD pipelines of Github/Azure DevOps, Jenkins (and many more) that simple forms can be added to trigger with parameters.

Why is it needed?

• Adding missing functionality allowing user friendly trigger of workflows/DAGs w/o specific expertise of DAG conf needed (learning curve)
• Also such entry form could provide an easy path to validate parameters before triggering (comparing to try to trigger with any JSON and then the DAG hits the wall because of bad parameters passed if user is not expert)
• Do not scare users when showing how to (self-service) workflows
• Prevent the need to add custom UIs in front of Airflow to "hide" Airflow internal complexity

Are there any downsides to this change?

• Airflow Codebase will be extended by ~1000 LoC (additional potential maintenance)
• Extension of UI must be discussed in conjunction with ongoing discussion and implementation of AIP-38 Modern Web Application

Which users are affected by the change?

Standard and power users of Airflow are not affected, there is mainly a very positive option possible for rare or entry users allowing to trigger DAG w/o mus experience, lowering the entry complexity.

How are users affected by the change? (e.g. DB upgrade required?)

With the proposed change no user or integrator is affected, the proposed change is just an extension w/o breaking change and no need to change DB layout

Other considerations?

Current workarounds used in multiple places are:

1. Implementing a custom (additional) Web UI which implements the required forms outside/on top of Airflow. This UI accepts user input and in the back-end triggers Airflow via REST API. This is flexible but replicates the efforts for operation, deployment, release as well and redundantly need to implement access control, logging etc.

2. Implementing an custom Airflow Plugin which hosts additional launch/trigger UIs inside Airflow. We are using this but it is actually a bit redundant to other trigger options and is only 50% user friendly

Open Questions needed to be discussed and decided before moving ahead?

• How does this extension relate and runs into a "potential dead end" of FAB if AIP-38 Modern Web Application is implemented in parallel?
  
  • If future UI is implemented in React, how can a custom UI be described? (Quote: "This likely calls for a POC implementation of a few options and choosing the best one based on a POC concept first. This is important, because it might heavily influence on the ways how such custom trigger ui will be described (declarative? Python code? Javascript? Pythonscript / webassembly maybe ? ) and deployed.")
    Note: Current proposal is made to provide Python classes to describe the UI (to be discussed of course), from Py code any YAML/Dict/whatever structure could be generated anyway. For the today's plugin we use YAML/Dict to define. But in DAG code Python classes might be more convenient also to have code assist?
  • When migrating to React: "how will we handle already existing plugins when migrating to a different implementation for these extensions ? An implementation with python code has the advantage of smoothing the migration process."
  • And as a direct follow-up: Will AIP-38 need to be completed BEFORE making this? Will this be made in parallel and then needs to be migrated over? Can the described concept be included in AIP-38 from the beginning? (First hand w/o customizations would be 90% already)
• Shall we make a PoC implementation in form of a PR for the discussion?
• Deployment: "security is super-important" - we need to know what will be the deployment model of those "customisations" - "who will be able to add them, how, what are the risks involved, whether ther will be any sandbox the custom code will be executed? Will it be server-side as well as client-only, or maybe it should only execute in the browser's javascript). I think there are a lot of "infrastructural" qustions to answer"
  Note/Partial Answer: Currently I would assume "standard" forms are with DAG development following any deployment/securty considerations as there is already potential to run/inject any arbitrary python code. Following same problems. For "custom" UI components it should follow the same approach like current Airflow Plugins. I'd propose to use FAB endpoints in a plugin and so this follows the same "problems" and challenges there are today. I do not want to add further attack points with this AIP. But might be related to question #1, if React will/shall be used.
• General question: "how UI customization should look like in a modern web app environment?" (Note: "modern" meaning "React")
• "ease of use for the end user is also a critical point (it should remain quite easy to extend the UI)" - How will this be achieved with this implementation?

What defines this AIP as "done"?

• Airflow delivers the option "out of the box" to define at least one form (per DAG) with simple input fields to trigger a DAG via UI and passing standard scalar parameters, e.g. string values
  
• Such forms can also be extended via UI plugins so that custom functionality can be directly linked into the standard UI "trigger DAG" buttons and allow replacing the standard dialogs w/o parameters or JSON dict entry