Versions Compared

Old Version 1

changes.mady.by.user Unknown User (maximebeauchemin)

Saved on

compared with

New Version Current

changes.mady.by.user Jarek Potiuk

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

 Authored by George Ke with Maxime Beauchemin on 2016-07-26

Overview

 

We want to add the ability to impersonate a unix user for a subprocess run.

Motivation

We need feature parity with Chronos tasks for the upcoming Chronos migration at Airbnb.

...

Additionally, we want to keep in mind a long-term goal of having a system that manages secrets that Airflow and its tasks need. We want to have the config & secrets that Airflow needs to be completely separate from the config & secrets operators need. Ideally, we would have a vault that manages secrets and provides what's needed to a user with a lease attached to it.  
 

Solutions

There have been many solutions proposed, listed below in chronological order. 

...

      • Also you can still view the secrets in the fork model (e.g. just have your task print the secrets to the log), so I wouldn't say this is a concern, it's actually more secure than forking the way I see it (since the task is run in complete isolation and memory space than the parent task).

Other Considerations

  1. Right now all airflow boxes are run as `airflow` user. To do impersonation, we need to run as a `sudo`er. This means that all existing jobs will be run as `sudo`er instead of `airflow`. This avoid this, we can allow the scheduler to be run as a `sudo`er but lower down to `airflow` before we run any tasks. This may either require a default "lower user" in `airflow.cfg` or having all operators set `unix_impersonate` so that it they don't run as `sudo`.

Conclusion

We are strongly leaning towards Solution 5 as it solves our following needs: 

...