Status 

Current state:  Voting Accepted

Discussion thread: here 

Vote thread: here 

JIRA:

...

The core issue that this KIP addresses is the non-idempotency of the initial heartbeat in the new consumer group protocol.If the member ID is null or empty, the server will reject the request
with an InvalidRequestException. 
When an AsyncKafkaConsumer sends a join group heartbeat but closes before receiving the response, it may attempt to send a leave group heartbeat without a member ID, leading to a failure with UNKNOWN_MEMBER_ID. This scenario can result in the broker registering a new member for which it will never receive a proper leave request.
Moreover, if the response to the initial heartbeat is lost, the client will retry the request, potentially creating a new member each time. This can lead to the creation of several "ghost" members that will persist until their session timeout elapses. During this period, these ghost members continue to hold onto partitions, causing delays in partition reassignment and inefficiencies in group management.

...

In the new version of the ConsumerGroupHeartbeat RPC, the client must generate a UUID as the member ID during the initial heartbeat. This member ID must be included in every subsequent request to ensure consistency. We highly recommend that users utilize a UUID as the member ID, but ultimately, the choice is up to the user. The server will validate that a valid member ID is provided in the member ID field. If the member ID is null or empty and the request is made over RPC version >= 2, the server will reject the request with an InvalidRequestException.

Lastly, to accommodate these updates, we also propose bumping the ConsumerGroupHeartbeat RPC from version 0 1 to version 12. This version upgrade reflects the new requirement for client-generated member IDs. It's important to note that despite these behavioral changes, there are no modifications to the existing fields themselves.

...

The consumer instance must generate a member ID when it starts, and this ID should remain consistent for the duration entire lifetime of the consumer's session. Here, a "session" is defined as the period from the consumer's first heartbeat until it leaves the group, either through a graceful shutdown, a heartbeat timeout, or the process stopping or dying. The consumer instance should reuse the same member ID for all heartbeats and rejoin attempts to maintain continuity within the group.process. The member ID acts as an incarnation ID of the process and should not be reset or changed, even if the consumer leaves and rejoins the group. It must remain the same until the process is completely stopped or terminated. 

Duplicate Member ID

Since we allow the client to generate the member ID themselves, duplicate member IDs might be a concern. However, as mentioned in the Motivation section, with improvements in third-party libraries like librdkafka , UUID generation has become easier today. We believe it is not difficult for the client to generate a unique ID, and moreover, the scope is limited to the consumer group. Therefore, we believe the risk of member ID collision within a group is negligibleIf a conflict arises where the member ID generated by the client is detected to be a duplicate within the same group (for example, the same member ID is associated with another active member in the group), the server will handle this by comparing the memberEpoch values of the conflicting members. The member with the lower memberEpoch is considered outdated and will be fenced off by the server. When this occurs, the server responds with a FENCED_MEMBER_EPOCH error to the client, signaling it to rejoin the group with the same member ID while resetting the memberEpoch to zero. This ensures that the client properly resynchronizes and maintains the continuity and consistency of the group membership.

Compatibility, Deprecation, and Migration Plan

...

• Retry Mechanism
  • Ensure the client retries with the same member ID if a heartbeat request fails.
• Consistency
  • Test that the client maintains the same member ID throughout the entire session and doesn't generate a new member ID midway.
• Error Handling
  • No member ID is provided by the client: The server should reject the request with an InvalidRequestException and log an error indicating the missing member ID.A member ID mismatch occurs within a session: If the server detects a mismatch between the provided member ID and the expected member ID for an ongoing session, it should return a UNKNOWN_MEMBER_ID  error.
• Backward Compatibility
  
  • Ensure that clients using older versions of the RPC (where the server generates the member ID) still function correctly.
• Mixed-Version
  • Test scenarios where multiple clients, using both the new and old versions RPC, are communicating with the server.

...