Better integration with Apache Kafka

Goals

• More fully leverage the power of Apache Kafka and provide seamless interaction between the modernized messaging mechanism that it represents and the managed dataflow platform that NiFi represents.

Background and strategic fit

Apache Kafka offers some compelling strengths including distributed durability, partitioned topics, and the general messaging paradigm itself which is ideal for the truly pub/sub use cases.  The current integration with Apache Kafka is fairly trivial with simple GetKafka and PutKafka processors.  To fully take advantage of some of the fundamental awesomeness of Kafka we should consider its use as an inter-nodal exchange mechanism, as a potential implementation of data queue, and as a more dynamic datasource such as automatic identification and selection of data from all topics.  What precisely should be done needs further exploration and we should discuss with the Kafka community what they think makes sense as well.

Assumptions

Requirements

User interaction and design

Questions

Below is a list of questions to be addressed as a result of this requirements document:

Not Doing