Page History

...

To determine the hash range before each for every consumer generation stabilizes, in the group assignment phase the leader will evaluate the situation and decide whether to trigger the sharing mechanism. For the very first version, a very intuitive criteria is comparing the relative size of topic partitions vs number of consumers that are opting into the key sharing mode. For example if number of consumers m > number of partitions n, we would do the assignment based off partitions instead of consumers. If 5 consumers subscribing to 1 topic with 3 partitions, the final assignment would be:

...

The new assignment comes from the fact that partitions are playing a reverse mapping to consumers. So in partition perspective, our assignment looks like a round robin assignment based off partitions:

tp1: M1, M4
tp2: M2, M5
tp3: M3: M1, M4
tp2: M2, M5
tp3: M3

The assigned ranges could then be used by the consumer to make their fetch more fruitful. We don't plan to introduce a separate assignor strategy, but instead we would like to rollout this strategy as a plugin to the existing assignment assignors. As of today the ConsumerPartitionAssignor interface looks like below:

public ConsumerPartitionAssignor {
  public Map<String, List<TopicPartition>> assign(Map<String, Integer> partitionsPerTopic,
                                                    Map<String, Subscription> subscriptions)
}

Take round robin assignor for an example, 

Restriction Enforcement

To satisfy different administration requirements, we are placing several configurations on:

1. Broker level, to determine whether to allow IC request,
2. Topic level, to determine whether this topic is allowed to be cooperatively processed 
3. Consumer level, to determine if this consumer opts to share processing with someone else

Merge restriction with above algorithm, 

The assigned ranges could then be used by the consumer to make their fetch more fruitful. This step unblocks the potential to allow a dynamic consumer group scaling beyond partition level capping.

...

Topic config

Consumer configs

...