Page History

...

The assigned ranges could then be used by the consumer to make their fetch more fruitful. This step unblocks the potential to allow a dynamic consumer group scaling beyond partition level capping.

Supporting Kafka Streams Rebalance

Kafka Streams have built a customized partition assignment semantics. For stateful DSL operations, there is a subtle requirement to make sure that the change in key distribution under the same partition will not encounter `key not found` error on the stream instance state. An example is like two stream instances A, B subscribing to one topic partition and do a per key aggregation. Record key m belongs to instance A for the first generation, however gets reassigned to instance B in the second generation. How does instance B know the previous states of record m? The answer is to replay the changelog topics. So in order to make Streams work with IC semantics, we could not split the stream changelogs because that would make it hard to maintain the key mapping. If multiple stream threads subscribing to the same partition, the changelog shall not be split. When the stream thread realizes a key range changes during rebalance, the restore consumer shall apply the new standalone mode key range assignment where the key based filtering rule could also be applied during restoration.

One more detail on stream is about task definition.

Look into the future: Transactional Support

...