...

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

KIP-848 introduces a next-generation consumer rebalance protocol that supports rack-aware partition assignment. In its initial implementation, the group coordinator computes subscribed topic metadata, which includes the topic UUID, name, number of partitions, and the rack set for each partition. When this metadata expires, the group coordinator recalculates the subscribed topic metadata and compares it to the current version. If there are differences, the group coordinator increments the group epoch and generates a new target assignment, effectively triggering a rebalance. However, the rack set for each partition consumes significant memory. In KAFKA-17578, a real-world case shows that for a group with 500 members and 2,000 topic partitions, partition rack data accounts for 79% of the total memory used by the ConsumerGroup object.

This KIP proposes removing the number of partitions and rack set details from the metadata and replacing them with a subscribed topic hash. Each topic is assigned a unique hash value. The topic hash must account for two server-side conditions that can trigger a rebalance:

• A topic with a new partition.

• A topic partition undergoes a rack change. Each topic partition has multiple replicas. The rack value is derived from the broker's broker.rack configuration, which is read-only and only changes when the broker restarts. If a broker stops, its associated replica is removed from the topic partitions. Conversely, when a broker starts, its replica is added to the topic partitions.

...

The PartitionRegistration also contains other changes like leader / elr. Every change will make the topic hash recomputation. It's no harm, because the hash only reflects change about uuid / name / number of partition / racks of partitions. If non of these fields change, the final hash will be the same, so it doesn't bump the group epoch or trigger a rebalance.

...

A single hash of all subscribed topic

...

in Group

The group doesn't need to store subscribed topic metadata, because the coordinator doesn't need the value to detect a rebalance. After this KIP, the group gets all subscribed topic hash from the cache in coordinator and sums them as a single hash. The final hash will be stored with a new group epoch in ConsumerGroupMetadataValue / ShareGroupMetadataValue.

...