DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
| Page properties | ||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
...
- Provide High Availability of Manager
- Primary Node Failover (HA) / Incorporate Leader Election functionalities
- Distributed State for user of extensions
- Rolling restarts and upgrades
- Provide multiple tiers of NiFi clusters
- Dynamic node registration, support for dynamic scaling of worker nodes
Management of data partitions among nodes in the cluster to allow for data affinity and allocation of tasks
- Background and strategic fit
Given the genesis of NiFi, clustering was designed to be extremely conservative in the interest of exactly once semantics and guarantee of avoiding data loss. While it is important to maintain this set of functionality, it also desirable to support other use cases where speed and volume are paramount to dataflow and processing with the caveats of eventual consistency and possible data duplication. State of the art for these scenarios is typically heavily leveraging ZooKeeper through a library like Curator.
...
| # | Title | User Story | Importance | Notes |
|---|---|---|---|---|
| 1 | Provide leader election | Need a robust mechanism as a basis for these extensions | ||
| 2 | Distributed State | Extensions need a consistent mechanism for sharing state | ||
| 3 | Dynamic Scaling | Support for worker nodes to join/leave cluster dynamically |
User interaction and design
...