Status

Current state: Under Discussion Accepted

Discussion thread: https://lists.apache.org/thread/l8ko353v3nn1blgymsty895x6c98oxlx

Vote thread: https://lists.apache.org/thread/j90htmphjk783p697jjfg1xt8thmy33p

JIRA:

...

A topic hash represents topic id, name, number of partitions, and partition racks. To avoid useless rebalance, the hash function should return same value for same data, even if it runs on different JDKs or partition racks have different order. For different JDKs, the KIP will use Murmur3 to compute the hash value. For partition racks with different order, we will compute hash for each value in it and sum as a result. We will also set the first byte as magic byte to represent hash version. The Murmur3 uses bit operation to ensure avalanche effect. A single bit change can get a different hash value.

...

...

The hash function should follow the order to combine different fields:

• magic byte (byte)
• topic id (long)
• topic name (string)
• partition size (long)
• sorted partition by id
  • partition id (int)
  • sorted racks (string)

SubscribedTopicDescriberImpl

...

A new metadata image is computed by controller. An epoch can represent the version of TopicImage. If there is number of partitions or partition racks change, the controller bumps the epoch. When coordinator receives a new metadata image and there is a new topic epoch, it triggers a rebalance for a group. The group also can store topic epoch map in records, so it can know the difference if the coordinator restarts. This approach save cpu and storage resources and avoid the downside of "Check Topic Delta to Trigger a Rebalance". However, this approach mixes the group coordinator logic within the controllerTrigger a Rebalance". However, this approach mixes the group coordinator logic within the controller.

Bump Group Version

Increase the group.version feature level to 2 to enable rack-aware rebalancing. However, there are two drawbacks to implementing this:

1. It requires maintaining lot of if-else block to determine whether to use subscription metadata or group hash.
2. When changing the group.version to 2, it triggers an epoch bump for all groups.