You are viewing an old version of this page. View the current version.

Compare with Current View Page History

Version 1 Next »

Status

State: Draft

Discussion thread:

JIRA:


Motivation

Operations performed on airflow server need to be run by a CLI which interacts directly with the database. This complicates authentication and interacting with the airflow installation by requiring the CLI to have a database user, rather than a web user.

Requirements

  • Reduce complexity in handling authentication.

  • Provide a layer of abstraction using an industry standard interface to allow authenticated, remote control for Airflow installations

  • Any API should confirm to existing industry standards

  • API structure should be discoverable

  • API should be versioned to handle any future backwards incompatible schema changes.

Proposal

Develop an a JSON restful API using the existing Plugin interface, defined by OpenAPI 3.

Implementation

API Definition

The API will be defined by a YAML OpenAPI 3 definition, which will be exposed via connexion. For the time being, the API JSON data structures will follow the existing definitions as defined in https://airflow.apache.org/api.html.

This means that client libraries can be autogenerated meaning development work is focused on the API structure and method handlers themselves, and not the infrastructure or client libraries. This also opens airflow up to being easily controlled by many languages and interfaces with minimal work required on the server side.

The API base url will have the format {protocol}://{airflowHost}/api/v{apiVersion}/, where:

  • protocol is one of http or https.

  • airflowHost is AIRFLOW__WEBSERVER__BASE_URL.

  • apiVersion is a integer.

Endpoints

For the initial work, the existing endpoints should not be modified, so that a minor release is not required to get this structure into the code base

The base url will prefix all endpoints. There will be significantly more endpoints to cover airflow functionality, but this list covers the existing experimental with some extra intermediary endpoints.

  • /dags

  • /dags/{dag_id}

  • /dags/{dag_id}/dag_runs (existing)

  • /dags/{dag_id}/dag_runs/{execution_date} (existing)

  • /dags/{dag_id}/dag_runs/{execution_date}/tasks

  • /dags/{dag_id}/dag_runs/{execution_date}/tasks/{task_id}

  • /dags/{dag_id}/status (existing as /dags/{dag_id}/paused/{paused})

  • /dags/{dag_id}/tasks

  • /dags/{dag_id}/tasks/{task_id}

  • /dag_runs (existing as /latest_runs)

  • /healthcheck (existing as /test)

  • /pools (existing)

  • /pools/{pool_name}

Discovery

To make the API discoverable, connexion provides a self-documenting API UI with swagger. Further to this, the API structure itself, should be discoverable implementing a concept such as HATEOAS via HAL.

Each resource or resource listing response should be defined in a structure defined in http://stateless.co/hal_specification.html.

NOTE: The exact structure of _links, and _curies are still to be defined.

Proof of concept

There is an example here of this implementation here https://github.com/apache/airflow/pull/4640/files

Important points to note are:

The sample documentation can be seen here:


Considerations

Using a structured definition like OpenAPI may restrict edge cases for complex, or non-json or non-rest based endpoints in the API.

Complex authentication methods may also be difficult to implement. The existing PoC above handles the authentication using the existing api authentication methods. In future, API keys or OAuth keys may be a better solution for API access, instead of requiring a session or an OAuth login.

  • No labels