Versions Compared

Old Version 2

changes.mady.by.user Colin McCabe

Saved on

compared with

New Version 3

changes.mady.by.user Colin McCabe

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

This KIP presents an overall vision for a scalable post-ZooKeeper Kafka.  In order to present the big picture, I have mostly left out details like RPC formats, on-disk formats, and so on.  We will want to have follow-on KIPs to describe each step in greater detail.    This is similar to KIP-4, which presented an overall vision which subsequent KIPs enlarged upon.

Overview

Image RemovedImage Added

Currently, a Kafka cluster contains several broker nodes, and an external quorum of ZooKeeper nodes.  We have pictured 4 broker nodes and 3 ZooKeeper nodes in this diagram.  This is a typical size for a small cluster.  The controller (depicted in orange) loads its state from the ZooKeeper quorum after it is elected.  The lines extending from the controller to the other nodes in the broker represent the updates which the controller pushes, such as LeaderAndIsr and UpdateMetadata messages.

...