Page History

...

KIP-848 introduces a next generation of consumer rebalance protocol which supports rack-aware partition assignment. In the first implement, the group coordinator computes subscribed topic metadata which contains topic UUID, name, number of partition, and rack set of each partition. If the metadata is expired, the group coordinator computes new subscribed topic metadata and compare with current one. If result is difference, the group coordinator bumps group epoch and calculates new target assignment which means it triggers a rebalance. However, the rack set of each partition takes too much space. In KAFKA-17578, a real case of memory usage shows that in a group with 500 members and 2K topic partitions,  partition racks account for 79% of whole ConsumerGroup object. This KIP will get rid of subscribed topic metadata and use a subscribed topic hash map to replace it. Each topic has a hash value. There are two server side conditions can trigger a new rebalance, so the topic hash needs to reflect them:

...