DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
...
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation
KIP-848 introduces a next-generation
...
consumer rebalance protocol
...
that supports rack-aware partition assignment. In
...
its initial implementation, the group coordinator computes subscribed topic metadata, which
...
includes the topic UUID, name, number of
...
partitions, and the rack set
...
for each partition. When
...
this metadata
...
expires, the group coordinator
...
recalculates the subscribed topic metadata and
...
compares it to the current version. If there are differences, the group coordinator
...
increments the group epoch and
...
generates a new target assignment
...
, effectively triggering a rebalance. However, the rack set
...
for each partition
...
consumes significant memory. In KAFKA-17578, a real-world case
...
shows that
...
for a group with 500 members and
...
2,000 topic partitions,
...
partition
...
rack data accounts for 79% of
...
the total memory used by the ConsumerGroup object.
This KIP
...
proposes removing the number of
...
partitions and rack set
...
details from the metadata and replacing them with a subscribed topic hash
...
. Each topic
...
is assigned a unique hash value.
...
The topic hash must account for two server-side conditions that can trigger a
...
rebalance
...
:
- A topic with a new partition.
A topic partition
hasundergoes a rack change. Each topic partition has multiple replicas
. Each replica is stored on a broker. The rack value is derived from the broker's
config. It's a read only config and can only be changed whenbroker.rackconfiguration, which is read-only and only changes when the broker restarts. If a broker stops,
a related replica will beits associated replica is removed from the topic partitions.
IfConversely, when a broker starts,
it will beits replica is added to the topic partitions.
Following table compares cpu / memory / disk usage of different strategies:
| CPU | Memory | Disk | |
|---|---|---|---|
1. Subscription topic metadata with topic UUID, name, number of partition, and rack set of each partition. | Low | High | High |
| 2. Cache mechanism and subscription topic metadata with topic UUID, name, and hash. | Mid | Mid | Mid |
| 3. A single hash to represent all subscribed topic per group. | High | Low | Low |
...
The second strategy
...
still requires significant disk space to store redundant data.
...
It stores a map of each subscribed topic hash. Even if only a single entry has changed, it results in storing a new map of topic hash. On the other hand, the third strategy heavily consumes CPU resources to recalculate the hash whenever a topic changes.
This KIP proposes combining
...
the second and third strategies.
...
The coordinator will maintain a cache to store
...
individual topic
...
hashes. When a topic
...
changes, only
...
its hash will be recalculated. Different groups can
...
retrieve the topic hash from the cache,
...
eliminating the need for the coordinator
...
to recalculate the same hash multiple times. In
...
the record, the coordinator
...
will aggregate all subscribed topic
...
hashes into a single hash per group and store
...
it on disk. This approach minimizes disk usage while efficiently handling changes.
Public Interfaces
ConsumerGroupPartitionMetadataKey / ConsumerGroupPartitionMetadataValue / ShareGroupPartitionMetadataKey / ShareGroupPartitionMetadataValue
...