Status 

Current state:  Voting Accepted

Discussion thread: here 

Vote thread: here 

...

The core issue that this KIP addresses is the non-idempotency of the initial heartbeat in the new consumer group protocol.If the member ID is null or empty, the server will reject the request
with an InvalidRequestException. 
When an AsyncKafkaConsumer sends a join group heartbeat but closes before receiving the response, it may attempt to send a leave group heartbeat without a member ID, leading to a failure with UNKNOWN_MEMBER_ID. This scenario can result in the broker registering a new member for which it will never receive a proper leave request.
Moreover, if the response to the initial heartbeat is lost, the client will retry the request, potentially creating a new member each time. This can lead to the creation of several "ghost" members that will persist until their session timeout elapses. During this period, these ghost members continue to hold onto partitions, causing delays in partition reassignment and inefficiencies in group management.

...

In the new version of the ConsumerGroupHeartbeat RPC, the client must generate a UUID as the member ID during the initial heartbeat. This member ID must be included in every subsequent request to ensure consistency. We highly recommend that users utilize a UUID as the member ID, but ultimately, the choice is up to the user. The server will validate that a valid member ID is provided in the member ID field. If the member ID is null or empty and the request is made over RPC version >= 12, the server will reject the request with an InvalidRequestException.

Lastly, to accommodate these updates, we also propose bumping the ConsumerGroupHeartbeat RPC from version 0 1 to version 12. This version upgrade reflects the new requirement for client-generated member IDs. It's important to note that despite these behavioral changes, there are no modifications to the existing fields themselves.

...

The consumer instance must generate a member ID when it starts, and this ID should remain consistent for the entire lifetime of the process. The member ID acts as an incarnation ID of the process and should not be reset or changed, even if the consumer leaves and rejoins the group. It must remain the same until the process is completely stopped or terminated. 

Duplicate Member ID

Since we allow the client to generate the member ID themselves, duplicate member IDs might be a concern. However, as mentioned in the Motivation section, with improvements in third-party libraries like librdkafka , UUID generation has become easier today. We believe it is not difficult for the client to generate a unique ID, and moreover, the scope is limited to the consumer group. Therefore, we believe the risk of member ID collision within a group is negligible.

Compatibility, Deprecation, and Migration Plan

...