Page History

...

In the proposed architecture, three controller nodes substitute for the three ZooKeeper nodes.  The controller nodes and the broker nodes run in separate JVMs.  The controller nodes elect a single leader for the metadata partition, shown in orange.  Instead of the controller pushing out updates to the brokers, the brokers pull metadata updates from this leader.  That is why the arrows point towards the controller rather than away.

Note that although the controller processes are logically separate from the broker processes, they need not be physically separate.  In some cases, it may make sense to deploy some or all of the controller processes on the same node as the broker processes.  This is similar to how ZooKeeper processes may be deployed on the same nodes as Kafka brokers today in smaller clusters.  As per usual, all sorts of deployment options are possible, including running in the same JVM.

The Controller Quorum

The controller nodes comprise a Raft quorum which manages the metadata log.  This log contains information about each change to the cluster metadata.  Everything that is currently stored in ZooKeeper, such as topics, partitions, ISRs, configurations, and so on, will be stored in this log.

...