Contact: fenglu@google.com, in collaboration with etune@google.com, daniel.imberman@gmail.com, wwlian@google.com, ramanathana@google.com

Motivation

The ongoing Airflow KubernetesExecutor discussion doesn’t have the story of binding credentials (e.g., GCP service accounts) to task PODs. Depending on how the kubernetes cluster is provisioned, in the case of GKE, the default compute engine service account is inherited by the  PODs created. It becomes a problem when users wish to attach different service accounts to a task POD. This document suggests a set of mechanisms to be incorporated into the Airflow KubernetesExecutor design so that any Airflow task can specify a set of credentials to be pre-configured on each task POD. We limit the scope of this document to GCP service account only, but our design can be seamlessly applied to other types of credentials (e.g., AWS access key).

...