Versions Compared

Old Version 3

changes.mady.by.user Unknown User (aldrin)

Saved on

compared with

New Version 4

changes.mady.by.user Unknown User (joewitt)

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  • Provide High Availability of Manager
  • Primary Node Failover (HA) / Incorporate Leader Election functionalities
  • Distributed State for user of extensions
  • Rolling restarts and upgrades
  • Provide multiple tiers of NiFi clusters

  • Management of data partitions among nodes in the cluster to allow for data affinity and allocation of tasks 

  • Background and strategic fit

Given the genesis of NiFi, clustering was designed to be extremely conservative in the interest of exactly once semantics and guarantee of avoiding data loss.  While it is important to maintain this set of functionality, it also desirable to support other use cases where speed and volume are paramount to dataflow and processing with the caveats of eventual consistency and possible data duplication.  State of the art for these scenarios is typically heavily leveraging ZooKeeper through a library like Curator.

...