...

The idea behind this KIP was initiated as a result of the discussion on the pull request for KAFKA-3664. Consumer assignment The original issue reported in the JIRA was about offsets of partitions not being committed before a consumer unsubscribes from them. Specifically, when users are using group management, if they call consumer.subscribe() or consumer.unsubscribe() to change the subscription, the removed subscriptions will be immediately removed and their offset will not be committed. The fix provided in the corresponding pull request includes performing a commitAsync() in subscribe() and unsubscribe() methods to trigger an offset commit only when auto commit is enabled for the consumer. This solution maintains the current invariants as far as consistency between the assignment and offset commits, and it addresses the main problem from the JIRA, which is basically that users will see duplicates when they change subscriptions with auto commit enabled. For users who are using manual commit, they will have to call commitSync() prior to changing their subscription, but that seems reasonable.

But if we consider the issue reported in the JIRA more carefully, we conclude that the root cause is consumer assignment that is currently updated immediately upon subscription changes (e.g. through this call, then this, and finally this). This behavior is something that can be improved due to following reasons (reference): 

• There is no known pattern for application development that would rely on the current behavior where the assignment is filtered immediately.
• The current behavior seems error prone with respect to resource cleanup since it is not unreasonable for a user to expect symmetric calls to onPartitionsAssigned and onPartitionsRevoked.
• The behavior today is inconsistent between subscriptions that are a list of topics and regex subscriptions: when you set a regex subscription, the assignment is not filtered immediately after the subscription change.
• Very few people probably rely on the intersection of functionality that would be affected: they would have to both pass in a ConsumerRebalanceListener, make subscription changes, and rely on the callback returning the filtered list.

Fixing this behavior would likely change the solution provided in the JIRA's pull request.

Public Interfaces

There are no new public interfaces or changes to existing interfaces suggested by this KIP.

...

The new consumer's implementation of topics subscribe and unsubscribe interfaces are modified so that they do not cause an immediate assignment update (this is how the regex subscribe interface is implemented). This is more about fixing an inconsistent behavior.

Compatibility, Deprecation, and Migration Plan

...