Page History

Status

Motivation

As user of Airflow for our custom workflows we often use DagRun.conf attributes to control content and flow. Current UI allows (only) to launch via REST API with given parameters or using a JSON structure in the UI to trigger with parameters. This is technically feasible but not user friendly. A user needs to model, check and understand the JSON and enter parameters manually without the option to validate before trigger.

Similar like Jenkins or Github/Azure pipelines we desire an UI option to trigger with a UI and specifying parameters. We'd like to have a similar capability in Airflow.

Considerations

This AIP was created after working a longer time with an UI plugin which we implemented customized allowing users to "easily" trigger DAGs with parameters. More and more we realize that every use case and demo we make requires an simple entry form to show "how easy" it is (not to scare users and have them running away). We propose to "productize" the feature from a plugin and merge it into the Airflow Core UI - hoping that more people can use this feature, having a lower entry barrier for "new" users and also to close gaps from standard UI path to trigger a DAG today compared to how the UI plugin is integrated.

We also had (in the past) a separate UI (outside of Airflow) just to collect parameters and trigger the Airflow DAG in the backend via REST API - but this was a full sepatration separation of the user from Airflow and at the end requires a very redundant UI being developed and maintained.

What change do you propose to make?

Implementation Part 1) Specifying Trigger UI on DAG Level

We propose to extend leverage the params attribute of the DAG class with an some additional optional attribute attributes so that an UI (s) (one or multiple form per DAG) can be specified in the DAG .

• Attribute name proposal: trigger_ui
• Type proposal: Union[TriggerUIBase, List[TriggerUIBase] (One or a list of UI definition inherited from an abstract UI class which implements the trigger UI)
• Default value proposal: [TriggerNoUI(), TriggerJsonUI()] (Means the current/today's state, user can pick to trigger with or without parameters)

With this extension the current behavior is continued and users can specify if a specific or multiple UIs are offered for the Trigger DAG option.

Implementation Part 2) UI Changes for Trigger Button

The function of the trigger DAG button in DAG overview landing ("Home" / templates/airflow/dags.html) as well as DAG detail pages (grid, graph, ... view / templates/airflow/dag.html) is adjusted so that:

1. If there is a single Trigger UI specified for the DAG, the button directly opens the form on click

2. If a list of Trigger UIs is defined for the DAG, then a list of UI's is presented, similar like today's drop-down with the today's two options (with and without parameters).

Menu names for (2) and URLs are determined by the UI class members linked to the DAG.

Implementation Part 3) Standard implementations for TriggerNoUI, TriggerJsonUI

Two implementations for triggering w/o UI and parameters and the current JSON entry form will be migrated to the new UI structure, so that users can define that one, the other or both can be used for DAGs.

Name proposals:

1. TriggerUIBase: Base class for any Trigger UI, defines the base parameters and defaults which every Trigger UI is expected to provide:

   • url_template: URL template (into which the DAG name is inserted to direct the user to)
   • name: Name of the trigger UI to display in the drop-down
   • description: Optional descriptive test to supply as hover-over/tool-tip)

2. TriggerNoUI (inherits TriggerUIBase): Skips a user confirmation and entry form and upon call runs the DAG w/o parameters (DagRun.conf = {})

3. TriggerJsonUI (inherits TriggerUIBase): Same like the current UI to enter a JSON into a text box and trigger the DAG. Any valid JSON accepted.

Implementation Part 4) Standard Implementation for Simple Forms (Actually core new feature)

Implement/Contribute a user-definable key/value entry form named TriggerFormUI (inherits TriggerUIBase) which allows the user to easily enter parameters for triggering a DAG. Form could look like:

   Parameter 1: <HTML input box for entering a value>
   (Optional Description and hints)
   
   Parameter 2: <HTML Select box of options>
   (Optional Description and hints)
   
   Parameter 3: <HTML Checkbox on/off>
   (Optional Description and hints)
   
   <Trigger DAG Button>

The resulting JSON would use the parameter keys and values and render the following DagRun.conf and trigger the DAG:

{
  "parameter_1": "user input",
  "parameter_2": "user selection",
  "parameter_3": true/false value
}

The number of form values, parameter names, parameter types, options, order and descriptions should be freely configurable in the DAG definition.

The trigger form should provide the following general parameters (at least):

• name: The name of the form to be used in pick lists and in the headline
• description: Descriptive test which is printed in hover over of menus and which will be rendered as description between headline and form start
• (Implicitly the DAG to which the form is linked to which will be triggered)

The trigger form elements (list of elements can be picked freely):

• General options of each form element (Base class TriggerFormUIElement:

  • name (str): Name of the parameter, used as technical key in the JSON, must be unique per form (e.g. "param1")
  • display (str): Label which is displayed on left side of entry field (e.g. "Parameter 1")
  • help (Optional[str]=Null): Descriptive help text which is optionally rendered below the form element, might contain HTML formatting code
  • required (Optional[bool]=False): Flag if the user is required to enter/pick a value before submission is possible
  • default (Optional[str]=Null): Default value to present when the user opens the form

• Element types provided in the base implementation

  • TriggerFormUIString (inherits TriggerFormUIElement): Provides a simple HTML string input box.
  • TriggerFormUISelect (inherits TriggerFormUIElement): Provides a HTML select box with a list of pre-defined string options. Options are provided static as array of strings.
  • TriggerFormUIArray (inherits TriggerFormUIElement): Provides a simple HTML text area allowing to enter multiple lines of text. Each line entered will be converted to a string and the strings will be used as value array.
  • TriggerFormUICheckbox (inherits TriggerFormUIElement): Provides a HTML Checkbox to select on/off, will be converted to true/false as value
  • TriggerFormHidden (inherits TriggerFormUIElement): Provides a pre-defined key/valure w/o displaying any form field, a.k.a HTML hidden form element

• Other element types (optionally, might be added later?) for making futher cool features - depending on how much energy is left

  • TriggerFormUIHelp (inherits TriggerFormUIElement): Provides no actual parameter value but allows to add a HTML block of help
  • TriggerFormUIBreak (inherits TriggerFormUIElement): Provides no actual parameter value but adds a horizontal splitter
  • Adding the options to validate string values e.g. with a RegEx
  • Allowing to provide int values (besides just strings)
  • Allowing to have an "advanced" section for more options which the user might not need in all cases
  • Allowing to view the generated DagRun.conf so that a user can copy/paste as well
  • Allowing to user extend the form elements...

Example view how today's solution is made via a UI plugin to integrate into Airflow UI (but is not linked into the standard "Trigger DAG" button):

Image Removed
Note that the advanced parameters and generated Job Config are hidden per default and need to be un-folded

Implementation Part 5) (Optional) Extended for Templated Form based on the Simple form but uses fields to run a template through Jinja

Implement (optionally, might be future extension as well?) a TriggerTemplateFormUI (inherits TriggerFormUI) which adds a Jinja2 JSON template which will be templated with the collected form fields so that more complex DagRun.conf parameter structures can be created on top of just key/value

itself (optional). As the params attribute is built upon the Params class to define meta-data to serve the purpose of JSON schema validations, this definition already serves 80% of details needed to render a form. See also https://airflow.apache.org/docs/apache-airflow/stable/concepts/params.html.

The following details to model a form can directly be leveraged from existing Params class definition:

• The order of fields of the form
• Name of the field, using the dict key. As this might not be very descriptive the existing JSON schema field title can be used if available for a better "Human readability.
• Type of the field derived from the JSON schema type and formatting clause
  • Type boolean can render a checkbox or on/off control
  • Elements of JSON format date and date-time can render a date picker
  • Type of arrays can be made as lists of text string and object types as a JSON sub-element entry
• Elements defined with an enum can be used to define drop-down selects for a pre-defined list op options
• If JSON schema allows to have types of null can control if a field is optional or required.
• The JSON schema attribute description can be used to render field help text
• Some JSON schema attributes for text like maxLength can be used to restrict user input on the form immediately
• Further attributes can be flexibly defined on Params class if we want to add form features in the future. Nevertheless available options and features can not be validated inside your editor as it is just extending kwargs. Therefore a later switch from FAB to a modern UI can be done w/o changing the structure f Form definition, just add a different UI logic.
• If just an arbitrary dict is used as default params then the existing dict value data types can be used to "guess" a suitable form even if the Params class for JSON schema is not used.
• We can use more attributes on the Params class also to allow rendering of custom HTML field

Some additional (optional) attributes are proposed to be added to control a form layout:

• An additional attribute hidden can be used to pass static values without populating to forms. Maybe not all values shall be presented to user.
• An additional attribute advanced can be used to have a user focusing on core elements and additional fields which are "expert options".

Nevertheless re-using the existing Params class comes with a set of limitations compared to a separate definition of a user entry form:

• Initial idea to allow definition of multiple forms per DAG is complex, re-using Params restricts to a single entry form per DAG (which might be acceptable to ease definition)
• Deep nested complex JSON schema (other than simple key/value) will be very complex to handle in a generic form so the proposed implementation would be limited to this. Otherwise any complex and optional deep nested JSON structure is killing any generic
• Usage of enum to define pick lists unfortunately restricts the user to add optional additional entries because JSON schema validation will kick-in.

With this extension the current behavior is continued and users can specify if a specific UI form is offered for the Trigger DAG option.

Implementation Part 2) UI Changes for Trigger Button

The function of the trigger DAG button in DAG overview landing ("Home" / templates/airflow/dags.html) as well as DAG detail pages (grid, graph, ... view / templates/airflow/dag.html) is adjusted so that per DAG the available form(s) can be controlled for better user guidance. Propsal is to add a DAG attribute trigger_ui:

1. If there is a single Trigger UI specified for the DAG, the button directly opens the form on click

2. If a list of Trigger UIs is defined for the DAG, then a list of UI's is presented, similar like today's drop-down with the today's two options (with and without parameters).

Menu names for (2) and URLs are determined by the UI class members linked to the DAG. The following options are proposed:

• "none": Skips any user entry form and triggers the DAG w/o further details using the default parameter values (like today's option "Trigger DAG")
• "json": Brings the user to the form where he can enter the JSON as dict like today with "Trigger DAG w/ config"
• "form": Shows the new form as described above as new feature in AIP-50
• a tuple with ("display": "url") can be used to provide a generic text label and a URL to forward the user to a custom plugin or UI as seamless integration if more complex UI is needed

DAGs not defining the attribute trigger_ui will have the options "none" and "json" like today plus if the DAG has params defined also a "form" ("form does only make a benefit if params are defined, else there is no base to render an empty (use-less) form)

Implementation Part 3) Standard Implementation for Forms (Actually core new feature)

Implement/Contribute a user-definable key/value entry form (general layout following todays JSON entry) which allows the user to easily enter parameters for triggering a DAG.

An example parameter set could look like

with DAG(
    dag_id="example_form_demo",
    (...)
    params={
        "param1": Param(
            "default value",
            type="string",
            title="Parameter1",
            description="(Optional Description and hints)",
        ),
        "parameter_2": Param(
            "user selection",
            enum=["foo", "bar", "user selection"],
        ),
        "flag3": Param(
            False,
            type="boolean",
            title="Parameter3"),
            description="(Optional Description and hints)",
        ),
    },
) as dag:
    run_this_last = EmptyOperator(
        task_id="run_this_last",
    )
    (...)

A form for this example could look like:

   Parameter1: <HTML input box for entering a value>
               (Optional Description and hints)
   
   Parameter2: <HTML Select box of options>
   
   Parameter3: <HTML Checkbox on/off>
               (Optional Description and hints)
   
   <Trigger DAG Button>

The resulting JSON would use the parameter keys and values and render the following DagRun.conf and trigger the DAG:

{
  "param1": "user input",
  "parameter_2": "user selection",
  "flag3": true
}

The number of form values, parameter names, parameter types, options, order and descriptions should be configurable in the DAG definition.

Example view how today's solution is made via a UI plugin to integrate into Airflow UI (but is not linked into the standard "Trigger DAG" button):

Image Added
Note that the advanced parameters and generated Job Config are hidden per default and need to be un-folded

Implementation Part 4) Examples

Provide 1-2 example DAGs which show how the trigger forms can be used. Adjust existing examples as needed.

Implementation Part

5) Documentation

Provide needed documentation to describe the feature and options. This would include an description how to add custom forms above the standards via Airflow Plugins and custom Python code. Proposal is to extend the docs section of "Params" (https://airflow.apache.org/docs/apache-airflow/stable/concepts/params.html) for the purpose of describing how to generate a form from the params definitions.

What problem does it solve?

With this proposal the functional gap is closed allowing to add user friendly trigger UI forms for staring DAGs with parameters (w/o the need that every user must understand the specific required JSON structure to trigger a DAG). Also it adds missing functionality which users know and expect to have from e.g. CI/CD pipelines of Github/Azure DevOps, Jenkins (and many more) that simple forms can be added to trigger with parameters.

Why is it needed?

• Adding missing functionality allowing user friendly trigger of workflows/DAGs w/o specific expertise of DAG conf needed (learning curve)
• Also such entry form could provide an easy path to validate parameters before triggering (comparing to try to trigger with any JSON and then the DAG hits the wall because of bad parameters passed if user is not expert)
• Do not scare users when showing how to (self-service) workflows
• Prevent the need to add custom UIs in front of Airflow to "hide" Airflow internal complexity

Are there any downsides to this change?

• Airflow Codebase will be extended by ~1000 LoC (additional potential maintenance)
• Extension of UI must be discussed in conjunction with ongoing discussion and implementation of AIP-38 Modern Web Application

Which users are affected by the change?

Standard and power users of Airflow are not affected, there is mainly a very positive option possible for rare or entry users allowing to trigger DAG w/o mus experience, lowering the entry complexity.

How are users affected by the change? (e.g. DB upgrade required?)

With the proposed change no user or integrator is affected, the proposed change is just an extension w/o breaking change and no need to change DB layout

Other considerations?

Current workarounds used in multiple places are:

1. Implementing a custom (additional) Web UI which implements the required forms outside/on top of Airflow. This UI accepts user input and in the back-end triggers Airflow via REST API. This is flexible but replicates the efforts for operation, deployment, release as well and redundantly need to implement access control, logging etc.

2. Implementing an custom Airflow Plugin which hosts additional launch/trigger UIs inside Airflow. We are using this but it is actually a bit redundant to other trigger options and is only 50% user friendly

Open Questions needed to be discussed and decided before moving ahead?

The initial idea of implementing forms as structured Python classes has been dropped by the feedback received to leverage the existing Params structure. Otherwise of course specific dedicated user entry form modules still can be added later. The proposed approach is a 80/20 proposal (gaining 80% of use cases direct benefit with 20% complexity).

Current workarounds known and used in multiple places are:

1. Implementing a custom (additional) Web UI which implements the required forms outside/on top of Airflow. This UI accepts user input and in the back-end triggers Airflow via REST API. This is flexible but replicates the efforts for operation, deployment, release as well and redundantly need to implement access control, logging etc.

2. Implementing an custom Airflow Plugin which hosts additional launch/trigger UIs inside Airflow. We are using this but it is actually a bit redundant to other trigger options and is only 50% user friendly as the standard "Run" buttons point to the JSON trigger dialog but not to the plugin and this is mis-leading from user perspective (not well integrate-able)

Open Questions needed to be discussed and decided before moving ahead?

•  How does this extension relate and runs into a "potential dead end" of FAB if AIP-38 Modern Web Application is implemented in parallel? From editors perspective the proposed structure is not bound to FAB and param structure can also be transferred to a new modern UI.
   How does this extension relate and runs into a "potential dead end" of FAB if AIP-38 Modern Web Application is implemented in parallel?
  
  •  If future UI is implemented in React, how can a custom UI be described? (Quote: "This likely calls for a POC implementation of a few options and choosing the best one based on a POC concept first. This is important, because it might heavily influence on the ways how such custom trigger ui will be described (declarative? Python code? Javascript? Pythonscript / webassembly maybe ? ) and deployed.")
    Note: Current proposal is made to provide Python classes to describe the UI (to be discussed of course), from Py code any YAML/Dict/whatever structure could be generated anyway. For the today's plugin we use YAML/Dict to define. But in DAG code Python classes might be more convenient also to have code assist?
  •  When migrating to React: "how will we handle already existing plugins when migrating to a different implementation for these extensions ? An implementation with python code has the advantage of smoothing the migration process."
  •  And as a direct follow-up: Will AIP-38 need to be completed BEFORE making this? Will this be made in parallel and then needs to be migrated over? Can the described concept be included in AIP-38 from the beginning? (First hand w/o customizations would be 90% already)
•  Shall we make a PoC implementation in form of a PR for the discussion?
•  Might be out-dated with Draft v2? → Deployment: "security is super-important" - we need to know what will be the deployment model of those "customisations" - "who will be able to add them, how, what are the risks involved, whether ther will be any sandbox the custom code will be executed? Will it be server-side as well as client-only, or maybe it should only execute in the browser's javascript). I think there are a lot of "infrastructural" qustions to answer"
  Note/Partial Answer: Currently I would assume "standard" forms are with DAG development following any deployment/securty considerations as there is already potential to run/inject any arbitrary python code. Following same problems. For "custom" UI components it should follow the same approach like current Airflow Plugins. I'd propose to use FAB endpoints in a plugin and so this follows the same "problems" and challenges there are today. I do not want to add further attack points with this AIP. But might be related to question #1, if React will/shall be used.
•  General question: "how UI customization should look like in a modern web app environment?" (Note: "modern" meaning "React")
•  "ease of use for the end user is also a critical point (it should remain quite easy to extend the UI)" - How will this be achieved with this implementation?

What defines this AIP as "done"?

•  Airflow delivers the option "out of the box" to define at least one form (per DAG) with simple input fields to trigger a DAG via UI and passing standard scalar parameters, e.g. string values
  
•  Such forms can also be extended via UI plugins so that custom functionality can be directly linked into the standard UI "trigger DAG" buttons and allow replacing the standard dialogs w/o parameters or JSON dict entry