Status

Current state: "Under Discussion"

Discussion thread: here

JIRA: here

Released: <Kafka Version>

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

The idea behind this KIP was initiated as a result of the discussion on the pull request for KAFKA-3664. Consumer assignment is currently updated immediately upon subscription changes (e.g. through this call, then this, and finally this). This behavior is something that can be improved due to following reasons (reference):

• There is no known pattern for application development that would rely on the current behavior where the assignment is filtered immediately.
• The current behavior seems error prone with respect to resource cleanup since it is not unreasonable for a user to expect symmetric calls to onPartitionsAssigned and onPartitionsRevoked.
• The behavior today is inconsistent between subscriptions that are a list of topics and regex subscriptions: when you set a regex subscription, the assignment is not filtered immediately after the subscription change.
• Very few people probably rely on the intersection of functionality that would be affected: they would have to both pass in a ConsumerRebalanceListener, make subscription changes, and rely on the callback returning the filtered list.

Public Interfaces

There are no new public interfaces or changes to existing interfaces suggested by this KIP.

Proposed Changes

The new consumer's implementation of topics subscribe and unsubscribe interfaces are modified so that they do not cause an immediate assignment update (this is how the regex subscribe interface is implemented). This is more about fixing an inconsistent behavior.

Compatibility, Deprecation, and Migration Plan

• Impacted users: Only those who currently rely on passing a ConsumerRebalanceListener, making a call to consumer's topics subscribe interface, and expecting an updated assignment in the callback are impacted. The number of users who rely on this specific use case is expected to be minimal to zero.
• If we are changing behavior how will we phase out the older behavior?
• If we need special migration tools, describe them here.
• When will we remove the existing behavior?

Test Plan

Describe in few sentences how the KIP will be tested. We are mostly interested in system tests (since unit-tests are specific to implementation details). How will we know that the implementation works as expected? How will we know nothing broke?

Rejected Alternatives

If there are alternative ways of accomplishing the same thing, what were they? The purpose of this section is to motivate why the design is the way it is and not some other way.