Leverage Local State to Do Temporal Analytics

Goals

• Enable Temporal Analysis using stateful processors

Background and strategic fit

The purpose of NiFi is to get the right data to the right place. Currently that is done in an effectively stateless manner by routing on the content and attributes of a single FlowFile. Now that NiFi has state for processors this opens up a whole new opportunity to leverage stateful processors to route data. An example of which is numeric anomaly detection using outliers. If a user only cares about the outliers generated by a sensor currently they have to route the data to another system to do streaming analysis before potentially routing the data back to a NiFi instance to finishing routing. With a stateful UpdateAttribute, users would be able to properly route just the outliers to their intended destination. 

This would open up many new use-cases in the IoT and Enterprise fields that currently have to be offloaded to streaming analytics platforms. 

Assumptions

• Until a more robust state provider that is designed to handle tens of thousands of requests per second is created, this will only use local state. 
• This will not be designed with the end-goal of acting as a streaming analytics platform but as another tool for the dataflow manager to route data.
  
  
• That decimals will be added to Expression Language  Unable to render Jira issues macro, execution error.  . Data analytics using Expression Language is very limited without the ability to work with decimals. That being said, there are a couple niche use-cases outside of data analytics that could use stateful analytics.

Requirements

User interaction and design

Questions

Below is a list of questions to be addressed as a result of this requirements document:

Not Doing