Page History

...

We store the following paths in Zookeeper:

1. Wiki MarkupStores the information of all live brokers.

   Code Block
   /brokers/ids/\[broker_id\] \--> host:port

   (ephemeral; created by admin)unmigrated-wiki-markup
2. Stores for each partition, a list of the currently assigned replicas. For each replica, we store the id of the broker to which the replica is assigned. The first replica is the preferred replica. Note that for a given partition, there is at most 1 replica on a broker. Therefore, the broker id can be used as the replica id

   Code Block
   /brokers/topics/\[topic\]/\[partition_id\]/replicas \--> \{broker_id …\}

   (created by admin) unmigrated-wiki-markup 
3. Stores the id of the replica that’s the current leader of this partition

   Code Block
   - /brokers/topics/\[topic\]/\[partition_id\]/leader \--> broker_id (ephemeral)

   (created by leader) unmigrated-wiki-markup 
4. Stores the id of the set of replicas that are in-sync with the leader

   Code Block
   - /brokers/topics/\[topic\]/\[partition_id\]/ISR \-->\{broker_id, …\}(created by leader)

5. This path is used when we want to reassign some partitions to a different set of brokers. For each partition to be reassigned, it stores a list of new replicas and their corresponding assigned brokers. Wiki Markup This path is created by an administrative process and is automatically removed once the partition has been moved successfully

   Code Block
   - /brokers/partitions_reassigned/\[topic\]/\[partition_id\] \--> \ {broker_id …\} (created by admin)

Key data structures

Every broker stores a list of partitions and replicas assigned to it. The current leader of a partition further maintains 3 sets: AR, ISR, CUR and RAR, which correspond to the set of replicas that are assigned to the partition, in-sync with the leader, catching up with the leader, and being reassigned to other brokers. Normally, ISR  AR and AR = ISR + CUR. The leader of a partition maintains a commitQ and uses it to buffer all produce requests to be committed. For each replica assigned to a broker, the broker periodically stores its HW in a checkpoint file.

...