Authored by George Ke with Maxime Beauchemin on 2016-07-26
Overview
We want to add the ability to impersonate a unix user for a subprocess run.
Motivation
We need feature parity with Chronos tasks for the upcoming Chronos migration at Airbnb.
Currently, all tasks are run as the same `airflow` user on a given worker. This makes it impossible to log who ran which task. With impersonation, commands run by operators can be ran as and logged as the user who wrote the task.
Security is also a concern, as right now all subprocesses have the same privileges as the parent process. Although we are not specifically tackling security, we want to make sure we don't regress or leave blatant security holes.
Additionally, we want to keep in mind a long-term goal of having a system that manages secrets that Airflow and its tasks need. We want to have the config & secrets that Airflow needs to be completely separate from the config & secrets operators need. Ideally, we would have a vault that manages secrets and provides what's needed to a user with a lease attached to it.
Solutions
There have been many solutions proposed, listed below in chronological order.
Solution 1: Simple
Sharing `airflow.cfg` and logging files with the user, (Link to commit) - Running the subtask command (`airflow run --raw`) with `sudo -u`
- user must be able to read `airflow.cfg` and be able to write to logging files
- Security concerns will allowing all users to read `airflow.cfg`
- `airflow.cfg` might be incomplete or not exist as we allow env vars as configuration
Solution 2: Pickles
Passing in a pickled subset of `airflow.cfg` to `airflow run --raw` (Link to commit) - The pickled subset config is passed as a command line argument
- You can also specify a custom logging directory on the operator level if the default logging files are not writable
- The `airflow run --raw` command will have secrets it, which can be seen with `ps`, in the syslog, as well as in the bash history
Solution 3: Forking
Forking the parent process and calling Python's `os.setuid` to change the user, (Link to commit) - A subset of `airflow.cfg` is passed into the forked process
- Only unix user is changed, so certain operators that use other indicators (e.g. smtp or hiveserver2) other than unix user for impersonation would implement them at an operator or hook level
- In terms of logging, the parent can pass file descriptors to the child that it can write to
- Subset of configs is safely passed to the subprocess without going through the command line
- A clean blanket solution that has a clear guarantee: that the Unix user will be set to the user passed in for the duration of the subprocess run
- Complexity/nuances with forked processes (e.g. shared file descriptors between parent and child)
- Security concerns. Because the forked process has the same memory as the parent, it has access to the `airflow.cfg` that's been loaded in memory. A bad python operator could read these secrets and/or dump them somewhere that the user can read. This makes code review as the gateway for enforcing security.
Solution 4: Operators
Operator and Hook level impersonation
- Add a `run_as_user` parameter on the `BaseHook`
- For each operator and associated hook that runs commands, we use `run_as_user` to do impersonation for each command in the operator/hook
- e.g. `sudo -u` for bash operator
- No secrets need to be passed to the subprocess as it has the same privileges as the parent. It's still secure as the commands that run in the operator are ran as the user
- Explicitly define what operators are used
- Hooks to APIs where impersonation isn't inferred from the unix username need this type of implementation regardless of the higher level solution.
- Making changes to all operators/hook with commands could be unwieldy and is specific to each operator/hook
- Things could be missed due to the large number of things being changed
- What the `run_as_user` parameter does could be foggy as operators do different things with it. There are not clear guarantees across operators.
Solution 5: Custom File
Parent process creates a temporary custom config file for the child to read.
- `airflow run --raw` for the child will be prepended with `sudo -H -u`
- The custom file will be a subset of `airflow.cfg` and only readable by the user. It will be deleted after the subprocess run.
- Better security than pickling. There will be no trace of secrets left in logs.
- Allows for an easier upgrade to using a vault of secrets when we decide to tackle that task (the custom file can be replaced with a call to the vault) as we are adding a codepath to isolate airflow secrets with operator secrets
- We're able to serialize the configuration object and get configuration from any source (env vars, ...)
- You can ssh into the machine and view the temp file
- Retort: allowing users to ssh into a machine is already insecure from a pure security standpoint. Also, you can create users on a machine that you cannot ssh as.
Other Considerations
- Right now all airflow boxes are run as `airflow` user. To do impersonation, we need to run as a `sudo`er. This means that all existing jobs will be run as `sudo`er instead of `airflow`. This avoid this, we can allow the scheduler to be run as a `sudo`er but lower down to `airflow` before we run any tasks. This may either require a default "lower user" in `airflow.cfg` or having all operators set `unix_impersonate` so that it they don't run as `sudo`.
Conclusion
We are strongly leaning towards Solution 5 as it solves our following needs:
We want clear guarantees for unix impersonation, implemented at the BaseOperator level and baked into the framework. The guarantee we want is simple: when using the `BaseOperator.unix_impersonate` (var name may change) the operator's execute method and hooks of a task instance will be executed as that user. Solution 5 is a blanket solution that fulfills this guarantee.
On top of that, individual hooks and operators that require impersonation beyond the Unix user itself should expose a `run_as` parameter with custom logic.
For instance someone maybe want the HiveOperator to run as the unix user, in which case `BaseOperator.unix_impersonate` would be applied. In another environment, someone may want to use the `HiveOperator.run_as` which essentially prepends the HQL command with a `SET proxy.user = {run_as}`. Both scenarios are legitimate, people may want either, and both will be possible with Solution 5.