Configuration Management of Flows

Goals

• Provide the ability for effective and automated configuration management of dataflows including merge, diffs, rollback across both the actual executing dataflow and all templates.

Background and strategic fit

Interactive command and control is a powerful feature and indeed helps organizations more rapidly establish and improve dataflows between systems.  This degree of freedom though in some environments creates new risks whereby an operator could mistakenly alter the behavior of the flow and be unable to quickly return it to its previous ‘good state’.  This is not a new problem area.  Configuration management of these flows can be thought of as being very similar to configuration management of source code.  NiFi should automatically be capturing and storing changesets and state as entities manipulate the dataflow.  Operators should be able to effect a rollback from the current state to a previous state by selecting the previous state and initiating a rollback.  The operator should also be able to see a visual indicator of the difference between the current state and the previous state.   In this sense changes could be seen as occurring along a linear chain/stack.  There may be merit in an even more complex branch/merge construct but the simpler case may be sufficient.

This same configuration management capability is also very useful for the management of flow templates as well.  Users should be able to establish versions for templates and be able to visualize the template and diffs between the various versions.

The version of a given dataflow or template also needs to take into account the availability and version of components within the flow.  That is to say we must consider what to do when a processor used within a version of the flow or a template is not present when applying that version again.  Should we fail on startup?  Fail to apply the template?  Should we set a ‘placeholder processor’ and prompt the user that they must select the new processor they’d like to use in its place and prompt them?

Assumptions

Requirements

User interaction and design

Questions

Below is a list of questions to be addressed as a result of this requirements document:

Not Doing