Status

Motivation

As we have been having conversations with Airflow users, who are using Airflow for Gen AI applications, there are a couple of features that have been brought up as being desirable. As context, based on the most recent Airflow survey, over 8% of Airflow users are now using Airflow for Gen AI use cases. 

One of the desired features is to have "human interaction" within the context of the overall orchestration flow. Here are the most common flows:

1. Choose a branch in the workflow: This is like a "human branch operator", where the human gets to pick one of the paths to move forward. This is a common pattern for business use cases such as content moderation, when the automated sentiment analysis is unsure if the content is "appropriate" and requires human judgement.  
2. Approve / Reject: This is a specialized form of the above, where the human approves or rejects the output from the prior tasks, thereby triggering a different branch in the flow. 
3. Provide input: The human is expected to provide textual input, which can then be used for subsequent steps. This is intended to be used within LLM workflows, where a human supplied context / guidance may be critical. 

For a lot of Airflow users, these operations above are expected to be performed through the Airflow UI. 

However, Airflow users especially those who use Airflow within the context of other applications may choose to build their own UX mechanisms for these human interactions. For those reasons, an API-driven interaction for the above workflows is also important. 

Though the urgency of the requirement for “human in the loop” interactions is driven by Gen AI applications, this need has existed in data pipelines earlier. A common use case within enterprise data pipelines has been for “human validations” to supplement “automated system validations”. 

Considerations

What change do you propose to make?

At a high level, we propose to make the following changes: 

• Introduce a set of “human operators” corresponding to the flows described above. 
• When tasks are created within a DAG using these operators, they are required to specify a role, which would be associated with one or more actual “humans” within the organization. The mapping of the role to the human is expected to be defined outside of the DAG and changeable based on the evolving needs of the organization. 
• The state of the individual task while waiting for human input is proposed to be “deferred”. However, this is TBD as we get closer to implementation, since there are potential tradeoffs.
• UI support:
• A user (with the appropriate role) can see the tasks waiting for their input. 
• The user can complete the task as appropriate by choosing one of the available options for that task, including providing input as needed. This is conceptually similar to the Trigger Dag form, but is tailored based on the operator. 
• In the case of a “branch” or “approve / reject” (i.e. flows 1 and 2 above), the Airflow UI will show the available options to the user. 
• In the case of the “provide input” (i.e. flow 3 above), the Airflow UI would display a form similar to that for DAG params today. 
• When multiple people have the same role, the first person to take action on a task “completes the task”. After that completion, other individuals cannot take action on that task and it is removed from their list of available actions. 
• API support: 
• While this task is waiting for this input, it can be queried using the API for details such as incoming Xcom data and available options. 
• This task can be marked as completed using the API, but requiring the key result as needed including the data needed for following tasks. 

Existing unchanged DAG / task behaviour of relevance:

• While waiting for human input, the status of the containing DagRun will be a regular valid Dagrun. 
• The tasks can use incoming Xcom data for the human to evaluate the operation and also supply outgoing Xcom data as the result of human intervention. The semantics here are unchanged from regular task handling. 

To be figured out:

• Notification options and settings for human oriented workflow systems tend to be customizable at an organizational level and at an individual level, rather than at a DAG / pipeline level. This is to be figured out for the right user experience and configurability / customizability.

Which users are affected by the change?

Current users of Airflow, using Airflow for existing use cases, will be unaffected by this change. 

How are users affected by the change? (e.g. DB upgrade required?)

Current users who don't use this new feature are unaffected by this change. 

Other considerations?

The only other AIP which is currently in progress which could be leveraged / impacted (though not necessarily) is AIP-68 Extended Plugin Interface. 

Human failure to respond:

• Clearly, humans are fallible and may not always be responsive. 
• Repeat notifications, escalations, timeouts for tasks waiting for human input is an important topic to consider.
• These processes exist today within organizations for such situations, but similarly to notifications, there is a high level of configuration and customization here.  

What defines this AIP as "done"?

Being able to create DAGs with the three flows described in the beginning of the document and run them through the Airflow UI and using the API (i.e. without the UI).