Status

Current state:  Under DiscussionAdopted

Discussion thread: TBD

JIRA: https://issues.apache.org/jira/browse/KAFKA-7610

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

...

In the current consumer protocol, the field `member.id` is assigned by broker to track group member status. A new consumer joins the group with `member.id` field set as UNKNOWN_MEMBER_ID (empty string), since it needs to receive the identity assignment from broker first. For request with unknown member id, broker will blindly accept the new join group request, store the member metadata and return a UUID to consumer. The edge case is that if initial join group request keeps failing due to connection timeout, or the consumer keeps restarting, or the max.poll.interval.ms configured on client is set to infinite (no rebalance timeout kicking in to clean up the member metadata map), there will be accumulated MemberMetadata info within group metadata cache which will eventually burst broker memory. The detection and fencing of invalid join group request is crucial for broker stability.

...

When encountering MEMBER_ID_REQUIRED exception, the client will use the given member id in the join group response to retry the join, which is expected to be accepted by the broker if id matches. If we encounter UNKNOWN_MEMBER_ID exception with the second join attempt, client handling logic will be the same, which is reseting the generation and ask a new member id from broker by sending anonymous join group request. We also handle the registered member id eviction through session timeout so that the pre-allocation map will not grow indefinitely, although the map size should be trivial since we only store a random generated id.

Effectively speaking, previously we accept anonymous member joining as new member, and now we require one more bounce to justify new member identity.

...