Proposal
The proposal is to extract the whole user management part of Airflow outside of core Airflow and introduce the user auth manager. The goal of the user auth manager is to manage all features and resources related to users, roles and permissions. This way users could simply chose between a very minimalist/simple user auth manager and a more advanced one with notion of groups/tenants. Everything under the FAB security manager as it exists today is extracted out from core Airflow and handled by the user auth manager.
Image Removed
Image Added
The user auth manager interface (or base user auth manager) is an interface each user auth manager needs to inherit from. This interface defines the common API of a user auth manager and is the only integration point with core Airflow. In other words, any action related to user management is done through classes inheriting from this interface.
Since it is impossible to forecast what feature/view each user auth manager is going to offer, the “Security” tab in the nav bar will be configured by each user auth manager.
User Auth managers are “pluggable”, meaning you can swap them based on your installation needs. Airflow can only have one user auth manager configured at a time; this is set by the userauth_manager
option in the [core]
section of the configuration file.
Implementations
In order to explain more into details how user auth managers work, I decided to take two different implementations of user auth manager:
- FAB user auth manager. This user auth manager offers the exact same features and experience as the current user management in Airflow. The implementation of this user auth manager is part of this AIP.
- KeyCloak user auth manager. This user auth managers leverages KeyCloak to manager users and roles. The implementation of this user auth manager is not part of this AIP. I still decided to include diagrams and explanations about this user auth manager in this AIP to increase clarity and understanding about the potential different implementations of user auth manager.
Minimalist FAB
user auth manager (backward compatible)
The target of the FAB user auth manager is to offer a backward compatible experience to the users. To put it simple, it moves the FAB security manager out of core Airflow to a new provider: the FAB provider. All the different pages are still served through the web server. The “Security” tab is configured to be as it is today. End users should see no difference between before user auth managers and after.
Image Removed
Image Added
KeyCloak
user auth manager
The target of the KeyCloak user auth manager is to delegate the user management to KeyCloak. The whole user management part is delegated to KeyCloak and admins have to configure roles and permissions in KeyCloak directly. A new provider KeyCloak needs to be created and contain only the KeyCloak user auth manager. This user auth manager will not be part of this AIP.
Image Removed
Image Added
Authentication flow
The authentication flow allows a user to log in Airflow. The flow follows the oauth 2.0 protocol.
![](/confluence/download/attachments/247827852/Authentication%20flow.png?version=4&modificationDate=1685477820000&api=v2)
When the authentication succeeds, a Flask session is created as it is today. This Flask session stores the user information about the user that is connected. Depending on the auth manager the kind of information stored can vary but, as an example, if the auth manager service used follows OpenID Connect (OIDC), the OIDC access token will be stored in the Flask session so it can be used anytime throughout the user session.
To simplify the example diagrams below, we consider the user is not logged in and the authentication on the backend side succeed.
FAB
user auth manager
FAB user auth manager is different from the other user auth managers. Instead of delegating the login experience to an external service, it includes and defines the login page within the manager. The page is still served through the web server. The goal is to have the login page as it is today.
Image Removed
Image Added
KeyCloak
user auth manager
Image Removed
Image Added
Authorization flow
The isThe is_authorized API API is the API each user auth manager needs to implement to check whether the current user has permissions to make a specific action. Here are some examples of usage:
- Has the current user permissions to list variables?
is_authorized([(permissions.ACTION_CAN_READ, permissions.RESOURCE_VARIABLE)])
- Has the current user permissions to read a specific DAG?
is_authorized([(permissions.ACTION_CAN_READ, permissions.RESOURCE_DAG)], "dag_id")
In order to understand how this API is implemented in different user managers, let’s take the use case of “User clicks on Variables in the Admin menu”.
Image Removed
FAB user manager
Image Removed
The is_authorized
API in the FAB user manager checks if the current user has the specified permissions. The implementation is really close to check_authorization in the security manager.
KeyCloak user manager
When logging in using KeyCloak, users are issued an access token stored in the metastore. This access token is used by KeyCloak to figure out if the current user has permissions to access a given resource.
Image Removed
Airflow Rest API
As part of the Rest API, some resources are no longer managed by core Airflow but by user managers: roles, permissions, users. As such, user managers needs to provide APIs to manage these resources. These APIs will be used by the Rest API to retrieve data from the user manager before returning it to the user.
Image Removed
One important requirement is, the Rest API schema (request and response) needs to be user manager agnostic. In other words, the format/schema of the Rest API request and response should be the same regardless of the user manager used. Having a Rest API schema different from one user manager to another would introduce breaking changes for users when changing user manager and also offer a non consistent experience across the Rest API. The idea is to return as part of the API all the attributes we know that are common across all user managers. Since it is hard to forecast what the future user managers will be able provide, my recommendation is to return a very minimalist set of attributes for each resource and an extra field to let each user manager return user manager specific attributes.
Roles
Current situation
This API provides as well some context about the action being made and the resource being accessed so that the auth manager can make some authorization decisions based on multiple parameters. This context needs to be extensible so that it is easy to add new information to it. You can find the schema of this context below. It is represented as JSON for readability purposes but since is_authorized
is a classic Python class method, they will be regular Python parameters.
Code Block |
---|
|
{
"nameaction": "stringPOST|GET|PUT|DELETE",
"actions "resource-type": [
{
"action"<resource-type>",
"resource-details": {
"name "id": "string"
},
"resource": {<resource-id>",
"tags": [<resource-tag-1>, <resource-tag-2>],
"name": "string"
}
}
]
} |
Proposal
There is no need to update the schema of roles. The schema looks generic enough and any user manager should be able to provide such information.
Permissions
Should we remove this resource from the Rest API?
Users
Current situation<other-resource-specific-information>,
},
} |
- action. Create (POST), Read (GET), Update (PUT) or Delete (DELETE) operation.
- resource-type. The type of resource being accessed. The different values of this parameter are the list of resources already defined in Airflow today (you can see this list in Security → Resources in Airflow UI).
- resource-details (optional). This is an optional extensible object to provide more metadata about the resource being accessed
- id (optional). The resource ID. If resource is "DAG", then this parameter relates to the DAG ID, if resource is "Variable", it relates to the Variable ID, etc...
- tags (optional). List of tags associated to the resource. Only available for resources you can tag, e.g. DAGs.
- This object is extensible, as such, other parameters can be added to this object to provide more information about the resource. e.g. "dag-folder", if resource is "DAG", specifies the DAG folder where the DAG is defined.
Examples:
Is the user authorized to create a new variable? | |
"first_namestring "last_namestring,
"username": "string",
"email |
Is the user authorized to read/list variables? | |
string "active": true,
"last_login": "string",
"login_count": 0,
"failed_login_count": 0,
"roles": [
{
"name": "string"
}
],
"created_on": "string",
"changed_on": "string"
} "resource-type": "Variable"
} |
|
Is the user authorized to read the variable "my-var-id"? |
Proposal usernamestring "emailstring "active": true,
"roles": [
{
"name "resource-details": {
"id": " |
|
string }
],
"created_on": "string",
"extra": {}
} |
Is the user authorized to delete the specific DAG "my-dag-id"? |
The field extra
contains extra parameters returned by user managers which are not common across all user managers. For instance, FAB user manager would return such data: usernamestring "emailstring "active": true,
"roles": [
{
"name "resource-details": {
"id": " |
|
string"
}
],
"created_onstring
extra": {
"first_name": "string",
"last_nameexample2"],
"dag-folder": " |
|
string "last_login": "string",
"login_count": 0,
"failed_login_count": 0,
"changed_on": "string",
}
}In order to understand how this API is implemented in different auth managers, let’s take the use case of “User clicks on Variables in the Admin menu”.
Image Added
FAB auth manager
Image Added
The is_authorized
API in the FAB auth manager checks if the current user has the specified permissions. The implementation is very close to check_authorization in the security manager.
KeyCloak auth manager
Image Added
Airflow Rest API
As part of the Rest API, some resources are no longer managed by core Airflow but by auth managers: roles and users. Therefore, these APIs will be removed:
However, some auth managers might need to define additional Rest API for their own needs. FAB auth manager is an example, in order to be backward compatible, the APIs listed above that are removed from core Airflow need to be redefined/moved from core Airflow to FAB auth manager. By default, no additional Rest API is defined in the base auth manager.
Airflow CLI
Among the sub-commands exposed by Airflow CLI, roles and users, similarly to the Rest API, need to fetch the data from user managers before returning the result to users. The plan is to use the exact same methods used by the Rest API to fetch the data from user managers.
User manager API
be removed from core Airflow. Like the Rest API, some auth managers might need to define additional CLI commands (e.g. FAB auth manager).
UI
The different UI pages used to manage users and roles are no longer part of Core Airflow and moved to auth managers. Depending on the auth manager and its service/tool used underneath, two options are possibles:
- Use the UI provided by the service/tool directly to manage users and roles. This is the preferred option.
- Create UI pages in the auth manager to manage users and roles. This is the option chosen for the FAB auth manager.
Even though the preferred option is to delegate entirely the user management to auth managers, the second option is necessary to implement the FAB auth manager.
Auth manager API
All auth All user managers have a common API defined in the user auth manager interface. You can find in the table below the common API needed from all user auth managers. The different categories are just for documentation and grouping purposes but might not be reflected in the architecture/code.
Category | Name | Description |
---|
UI | get_tab_title() | Returns the tab title in the nav bar. Currently "Security" |
---|
get_tab_menu() | Returns the different items when hovering the tab |
get_url_accounturl_user_profile() | Returns URL to access my account/user profile |
get_user_name() | Returns the user name |
Core | get_url_login() | Returns URL to sign in |
---|
get_url_logout() | Returns URL to sign out |
is_logged_in() | Return true if the current user is logged in |
postlogin_logincallback() | Post login operations Callback called after login. It might be needed depending on the user auth manager used. e.g. Storing the OIDC access token |
is_authorized() | Is the user authorized to make an action on a given resource. See section "Authorization API" for more details | Rest API/CLI | list_users() | List all users |
get_user() | Return user information |
create_user() | Create one user |
update_user() | Update one user |
delete_user() | Delete one user |
list_roles() | List roles |
get_role() | Return role information |
create_role() | Create one role |
update_role() | Update one role |
delete_role() | Delete one role |
get_security_manager_override_class() | Specific an override for the security manager. Depending on the auth manager, you might need to override the security manager to add custom logic (e.g. register specific views) |
Additional resources | rest_apis() | Define additional Rest APIs |
---|
cli_commands() | Define additional CLI commands |
Future work
Here are some examples of task that are not part of the AIP but can be done as follow-up once the AIP is completed.
- Create KeyCloak provider and KeyCloak user auth manager within it
- Additional providers (e.g. AWS user auth manager, Google user auth manager)
Considerations
What problem does it solve?
It makes user management component of Airflow pluggable and extensible by introducing a user an auth manager interface in the core Airflow that can be extended by any provider package who want to support user management natively. An extensible and pluggable user management would open up the potential for a more advanced user management features than there is today in Airflow such as group of users (or tenants). Why is it needed?
Having a user management which fits everyone needs (from individuals to teams within enterprises) is impossible. Users need to have an extensible and pluggable way to use and define the user management they want.
Native user management support in cloud providers means that roles can be mapped directed to identity provider. Currently, Airflow operators have to work around this and are unable to provide seamless RBAC in Airflow when running it on different cloud platforms.
Which users are affected by the change?
All users are impacted by the change. Though, by default Airflow would use the FAB user auth manager that is backward compatible and users should not see any difference. Of course, if an admin decides to change the user auth manager to use another one, then the whole user management experience of the environment would change.
What defines this AIP as "done"?
- The user auth manager interface defined
- New provider FAB provider created
- FAB user auth manager inhering from the user auth manager interface defined. This FAB user auth manager is part of the new provider: FAB provider. By default Airflow uses this user auth manager