Multi-Tentant Dataflow

Goals

• Provide the ability for multiple groups of entities (people or systems) to command, control, and observe state of different parts of the dataflow with varying levels of authorization.

Background and strategic fit

Within a single system Apache NiFi can support thousands of processors and connections, which translates to an extremely large number of dataflows for even the largest of enterprise use cases.  This in turn means each cluster of NiFi servers is capable of handling the requirements of one or more organizations.  However, the authorization model of NiFi today means that the authority level of a given dataflow applies to the entire dataflow graph.  This is not sufficient to support the multi-tenancy needs that are present when multiple organizations leverage the same resources to manage dataflows.

We should identify an appropriate design for multi-tenant authorization allowing administrators to establish authority per entity to a specific process group with reasonable transitivity to child groups.  Much of the plumbing to make this work already exists and the user experience around authorization should not require any drastic changes.  The types of process group specific authorizations should be granular enough to support read-only, changing dataflow behavior, starting and stopping components, viewing data provenance sourced to that group, viewing content sourced to or belonging in that group, etc..

Another approach may be to introduce the concept of different workspaces. Each workspace would act as a boundary for user authorization while providing the benefit of an additional canvas to design on. Workspaces could be created/removed dynamically (by a user with appropriate authority) allowing for the creation of visually separate dataflows. Additionally, this approach would allow each workspace to define root group ports for performing site to site communications. Of course, we would support sending data between workspaces locally. This may work similar to existing site to site connections. 

Assumptions

Requirements

User interaction and design

Questions

Below is a list of questions to be addressed as a result of this requirements document:

Not Doing