| Target release | Release name or number |
|---|---|
| Epic | Link to related JIRA epic or feature |
| Document status | |
| Document owner | |
| Designer | Lead designer |
| Developers | Lead developer |
| QA | Lead tester |
|
Being responsive to changing and evolving needs of data collection and aggregation requires changes to be adaptable to changing needs an organization has for the information being collected.
This is two faceted in terms of needed functionality. First, a user experience and interface for the designing and versioning of flows. The second, a means of making flows available for instances to receive causing updated processing to occur.
This could be an extension of the core NiFi interface but a separate workspace and feel. At its core, a minifi-api could be introduce which functions similarly to the nifi-api, a REST API that drives the user interface and core design functionality. The reason for a separate module is to allow arbitrary enabling/disabling of the MiNiFi functionality in a NiFi instance. While a similar user experience to NiFi in terms of design is extremely valuable, the context and palette available is very much discrete. To that end, the workspace approach would allow a separate context for users to carry out the task of managing their MiNiFi flows with unique tooling to that workflow.
Users could create flows on a per MiNiFi class basis. A class is defined as a group of MiNiFi instances that share a common flow. Using an approach similar to that content outlined with the Configuration Management of Flows.
Users would also be able to select the current, or active, flow for a given class of instances and make this available for deployment. At minimum, metadata would include a hash or signature of the flow as well as an identifier
The other scenario to be supported for MiNiFi is more application focused and provides the needed infrastructural components. At its core, this introduces a Command and Control API (C&C API) which is inherently a defined set of REST endpoints and resources that could be implemented in any language of choice. An initial implementation could be created in Java in a manner analogous to that of the aforementioned nifi-api and minifi-api modules.
An important note is the positioning and nature in which the C&C API could be deployed and utilized. As systems extend farther from core infrastructure and networking, the means by which communication occurs increases in complexity inclusive of items such as availability, bandwidth, NAT traversal and organizational and security policies. As a result, there may be varying tiers of access and the need for a common API to be available and consumable in a distributed and possibly localized manner. In NiFi environments, the idea of the Flow Persistence Provider could provide a façade to a more canonical repository of flows or cache and provide a subset of those flows locally.
Specific implementations of the C&C API could provide sophisticated provisioning of flows to subgroups of classes akin to split testing based upon individual MiNiFi instance metadata.
Flows could be consumed through various means driven by the Configuration Change Notifier/Listener approach currently provided in an initial implementation and design in the MiNiFi codebase. This allows MiNiFi to be amenable to the mechanism in which flows could be transferred to a given set of instances. The desired mechanism would be to make use of the C&C API directly, but in some cases may require a file to be delivered to a specific directory. While there may be advantageous paths as default means of transport, the C&C API in conjunction with extensible Configure Change Notifiers allows instances to be adaptable to realities of an organization’s network and compute infrastructure.
Making use of immutably versioned flows provided to instances would allow the tagging of FlowFile data and/or provenance events generated, tied to a specific flow version. This empowers the destination systems of MiNiFi data to make determinations on the inherent worth of the data received. For those instances where data is collected/generated by a system that has an outdated flow, it may be of little or no value or require additional/separate processing.
| # | Title | User Story | Importance | Notes |
|---|---|---|---|---|
| 1 | Short identifier for the story | Describe the user and what they are trying to achieve | Must Have |
|
| 2 |
Include any mockups, diagrams or visual designs relating to these requirements.
Below is a list of questions to be addressed as a result of this requirements document:
| Question | Outcome |
|---|---|
| (e.g. How we make users more aware of this feature?) | Communicate the decision reached |