...

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Currently, if an AsyncKafkaConsumer closes after sending The core issue that this KIP addresses is the non-idempotency of the initial heartbeat in the new consumer group protocol.
When an AsyncKafkaConsumer sends a join group heartbeat but closes before receiving the response, it may attempt to send a leave group heartbeat without a member ID. This results in , leading to a failure with UNKNOWN_MEMBER_ID. Consequently, This scenario can result in the broker ends up with registering a registered new member for which it will never receive a proper leave request.
More importantly, this scenario highlights that Moreover, if the response to the initial heartbeat is not idempotent. If a member is created on the first request but the response is lost for some reason, the client will retry the heartbeat request, leading potentially creating a new member each time. This can lead to the creation of a new member. If this happens multiple times, several “ghost” members could be created, which will only expire when several "ghost" members that will persist until their session timeout elapses. During this period, these ghost members will continue to hold partitions until they expireonto partitions, causing delays in partition reassignment and inefficiencies in group management.

The main downsides of this current logic are as follows:

...

These issues lead to inefficient partition management and potential delays in rebalancing, affecting the overall system performance and reliability. Therefore, it is crucial to address this behavior to ensure timely and accurate group membership management and partition rebalancing.

Public Interfaces

None.

Proposed Changes

By addressing the non-idempotency of the initial heartbeat, this KIP aims to resolve these issues, ensuring more accurate group membership management and timely partition rebalancing.

Last but not least, initially in KIP-848, we rejected the idea of allowing the client to generate its own Member ID due to concerns about additional dependencies and potential issues with correct ID generation. However, since then, improvements in libraries like librdkafka  have addressed these concerns. Given these changes, we now propose requiring client-side Member ID generation, as it simplifies the process and improves reliability.

Public Interfaces

We propose bumping the version of the ConsumerGroupHeartbeat RPC from version 0 to version 1 to reflect updates to the behavior regarding the member ID. In this new version, the To address the issue mentioned in the Motivation section, we propose bumping the version of the ConsumerGroupHeartbeat RPC from version 0 to version 1.
In the current version of the ConsumerGroupHeartbeat RPC, the client can generate and provide a member ID in the heartbeat request. If the client does not provide a member ID, the server will generate one and return it to the client.In the proposed new version of the ConsumerGroupHeartbeat RPC, while there are no changes to the RPC fields, the behavior regarding the member ID has been updated.
The server will no longer generate a member ID. Instead, the client must generate a UUID as the member ID during the initial heartbeat and include it in every subsequent request. The server will validate that a valid UUID is provided in the member ID field. If , and if the member ID is missing, the server will reject the request. It's important to note that despite these behavioral changes, there are no modifications to the fields themselves.

Proposed Changes

To implement the changes proposed in the KIP, the client will need to generate UUIDs. The simple specifications are as follows:

UUID Generation Specifications

1. UUID Version:

   • Use UUID version 4 for randomness and uniqueness, aligned with Kafka's implementation.

2. Uniqueness:

   • Ensure no generation of reserved UUIDs (e.g., all-zero UUIDs) to maintain uniqueness.

3. Encoding:

   • Represent UUIDs using base64 encoding for compactness and URL safety.

Compatibility, Deprecation, and Migration Plan

If there is The change is backward compatible because version 0 of the ConsumerGroupHeartbeat RPC already supports a member ID provided by the client. If a client using an older version of the ConsumerGroupHeartbeat RPC , the join flow sends a heartbeat, it will still follow the rules which are mentioned outlined in KIP-848#Member ID.
If newer version clients receive an older version heartbeat, they should still adhere to the new behavior, generating Newer clients should generate the member ID on the client side. This , as this approach is preferred because we believe client-side ID generation is more reliable and scalable than server-side generation, and there are no changes to the fields between the newer and older version RPCs. Ultimately, we aim to . Our goal is to eventually phase out server-side member ID generation entirely.

...

2.Server-side Generate the UUID at First Heartbeat but without adding the member to the group

Using In this approach, the server generates a UUID during the first Heartbeat sent out by the client to generate the UUID without adding the , but does not yet add the member to the group yet. The first Heartbeat would send back return the new member ID, zero as the a member epoch of zero, and zero as the a heartbeat interval of zero, prompting the client to immediately send another Heartbeat.

Reason for Rejection

This solution also introduces the complexity of backporting. When using the first Heartbeat to obtain a UUID from the server, compatibility issues must be addressed because the old server still adds the member to the group when handling the first Heartbeat. 
Making the first Heartbeat idempotent (i.e., generating the member ID only) represents a behavioral change. This means clients can no longer assume that the first Heartbeat will both "create a member ID" and "add the member to the group." Last and the least, this will need to update the content of KIP-848While this approach was rejected because if a client leaves the group after receiving the first Heartbeat and the member ID, the server would respond with an "unknown member ID" error since the member is not yet officially part of the group. Overall, the client-side generated member ID is a simpler and more reliable solution.