Page History

...

We want to clarify beforehand that this KIP would be a starting point of a transformational change on the consumer consumption semantics. It's not possible to have all the design details rolling out in one shot. Instead, the focus is to define a clear roadmap of what things need to be done, and better illustrate the long term plan while getting some concrete tasks in starting steps. There will also be follow-up KIPs and design docs, so stay tuned.

...

As stated above, the scaling cap for consumer based application is the number of input partitions. In an extreme scenario when there is one single input partition with two consumers, one consumer must remain idle. If the single box consumer could not keep up the speed of processing, there is no other solution to it but lagging. It would be ideal we could co-process data within one partition by two consumers when the partition level order is not required, such that we could add as many consumer instances as we want.

...

partitionsPerTopic := Map<String, Integer>
topicAllowKeyRangeFetch := Map<String, Boolean>
Subscriptions := Map<String, Subscription>
brokerAllowICcoordinatorAllowIC := Boolean

groupSubscriptionByTopic
Step one: group subscription by topics
Group consumers based on their topic subscription. Inside the value, we put partition count and consumer.ids inside
subscriptionByTopic := Map<String, Tuple<Integer, List<String>>>

NumberOfPartitionsfor topic, numPartitions, List of consumers:
	if numPartitions < number of consumers and coordinatorAllowIC and topicAllowKeyRangeFetch[topic]:
		consumersAllowKeyRange := sizeSet of inputTopicPartitionsConsumers who could allow KeyRange fetch and commit
		Assign consumers in a round robin manner to partitions

This step unblocks the potential to allow a dynamic consumer group scaling beyond partition level capping.

...