Page History

...

KIP-848 introduces a next-generation consumer rebalance protocol that supports rack-aware partition assignment. In its initial implementation, the group coordinator computes subscribed topic metadata, which includes the topic UUID, name, number of partitions, and the rack set for each partition. When this metadata expires, the group coordinator recalculates the subscribed topic metadata and compares it to the current version. If there are differences, the group coordinator increments the group epoch and generates a new target assignment, effectively triggering a rebalance. However, the rack set for each partition consumes significant memory. In KAFKA-17578, a real-world case shows that for a group with 500 members and 2,000 topic partitions, partition rack data accounts for 79% of the total memory used by the ConsumerGroup object. The rack-aware rebalance was removed in 4.0. This KIP introduces another way to support the feature and targets at 4.1

This KIP proposes removing the number of partitions and rack set details from the metadata and replacing them with a subscribed topic hash. Each topic is assigned a unique hash value. The topic hash must account for two server-side conditions that can trigger a rebalance:

...