...

This proposal would add a new option to existing assignment strategies of the new consumer. It would not impact any existing functionality.The only impact is to those new consumers that have some partition cleanup code in their onPartitionsRevoked() callback listeners. That cleanup code is rightfully placed in that callback listener because the consumer has no assumption or hope of preserving any of its assigned partitions after a rebalance when it is using range or round robin assignor. One of the advantages of the sticky assignor is that, in general, it reduces the number of partitions that are actually de-assigned from a consumer. Because of that, consumers now can do their cleanup more efficiently. Of course, they still can perform the partition cleanup in the onPartitionsRevoked() listener, but they can be more efficient and make a note of their partitions before and after the rebalance, and do the actual cleanup after the rebalance on the actual partitions they lost (which is normally not a whole lot). The code snippet below clarifies this point.

public class StickyAssignorConsumerRebalanceListener implements ConsumerRebalanceListener {


    ...
 
    private Collection<TopicPartition> revokedPartitions;
 
    @Override
    public void onPartitionsAssigned(Collection<TopicPartition> partitions) {
        this.revokedPartitions = partitions;
    }

    @Override
    public void onPartitionsRevoked(Collection<TopicPartition> partitions) {
        this.revokedPartitions.removeAll(partitions);
        cleanup(this.revokedPartitions);
    }
 
    ...
}

Rejected Alternatives

• Having the consumer group leader store the calculated topic partition assignment in an internal topic for other consumers to retrieve in case of a leadership change. It was decided that passing the calculated assignments as user data to all consumers after each rebalance is a more viable option.