DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
...
This KIP proposes adding a UUID called the client instance ID into the request header of all Kafka protocol requests. Those familiar with KIP-714 will be aware that it already introduced the client instance ID, so this KIP actually proposes elevating that concept to into a universal unique identifier for client instances in all RPCs.
...
The client instance ID is added as a tagged field in the RPC request header, so it is present on all RPCs that use the v2 request header, which is almost all of the current versions of the RPCs (only SaslHandshake and OffsetDelete use the v1 request header for their latest versions). The client is required to be consistent in its use of client instance ID in request headers. If it specifies the value in its request headers, every request on a connection must specify the same value. If it does not specify the value in its request headers, every request on a connection must not specified a value.
If a connection specifies a Note that omitting a client instance ID in from the request header of its first request which uses the v2 request header, it must specify the same and explicitly sending a zero UUID as the client instance ID in the request header are indistinguishable in the protocol and are considered semantically equivalent. This means that if a client explicitly sets a zero UUID, the broker will treat it as if the client had not set a client instance ID. When the following text says "does not specify a client instance ID", this includes specifying a zero UUID as the client instance ID.
If a connection specifies for all subsequent requests which use the v2 request header. The initial client instance ID for each connection will be cached by the broker for checking (this is an implementation detail, but caching it in the ChannelMetadataRegistry is an option). Once a client has specified a client instance ID in the request header of its first request, any subsequent requests which are missing the client instance ID (with the exception of requests using the v1 request header) or which specify a different value for the client instance ID will be rejected with error code INVALID_REQUEST.If a client does not specify a client instance ID in the request header of its first request which uses the v2 request header, it must not specify a the same client instance ID in the request header of any for all subsequent requests . If it does so, the request will be rejected with error code INVALID_REQUEST.
Alignment of identifiers
...
which use the v2 request header. The initial client instance ID for each connection will be cached by the broker for checking (this is an implementation detail, but caching it in the ChannelMetadataRegistry is an option). Once a client has specified a client instance ID in the request header of its first request, any subsequent requests which are missing the client instance ID (with the exception of requests using the v1 request header) or which specify a different value for the client instance ID will be rejected with error code INVALID_REQUEST.
If a client does not specify a client instance ID in the request header of its first request which uses the v2 request header, it must not specify a client instance ID in the request header of any subsequent requests. If it does so, the request will be rejected with error code INVALID_REQUEST.
Alignment of identifiers
By adding client instance ID to the request headers, we now had a unique application instance identifier which we can use for other purposes such as the member ID in the group protocols and the client telemetry client instance ID. The client MUST use the same UUID in request headers as it uses for the client telemetry client instance ID. The alignment of other identifiers is by convention (and the Java client will follow the convention) rather than mandate. In the language of standards, the client SHOULD use the same UUID in request headers as it uses for the member ID in the group protocols. This alignment just makes traceability and problem determination more straightforward.
In the modern group protocol RPCs such as ConsumerGroupHeartbeat and ShareGroupHeartbeat , the member ID is a string. In practice, it is a UUID which is encoded into a string, but the nature of this conversion is not specified in the protocol and it really is treated as a string in the broker. The Java client uses the org.apache.kafka.common.Uuid class to generate the member ID and convert it into a string. The client instance ID really is a UUID in the protocol. When the Java code in the broker converts this into a string, it uses the same Java code as the client does for the member ID. As a result, the member ID and client instance ID can trivially be the same when represented as strings, even though they have different data types in the protocol. For traceability and problem determination, this "conventional" alignment works well.
However, at least one other Kafka client implementation uses a slightly different string encoding of the member ID which does not match that generated by org.apache.kafka.common.Uuid . This is valid with regards to KIP-848 and KIP-932, but does unfortunately mean that the member ID does not look the same as the client instance ID, even if they have the same original UUID value in the client.
It would be possible to go around the existing RPCs such as ConsumerGroupHeartbeat and GetTelemetrySubscriptions , and remove the fields containing the existing identifiers which are intended to be aligned. Doing so would be a bad idea though, because we would then have RPC versions which essentially depend upon the presence of a tagged field in the request header. This is a protocol-compatibility nightmarea protocol-compatibility nightmare because tagged fields are optional and having a hard dependency on an optional field seems unwise.
This KIP makes one change to the GetTelemetrySubscriptions behavior. If the client requests The client may only request a new client instance ID on its initial GetTelemetrySubscriptions request (request.clientInstanceId is 0) and also sends , if it does not also send a client instance ID in the request header. After this KIP, the broker will send back the is only expecting to generate telemetry client instance ID from the header for use with client telemetry rather than generating a new UUIDfor older clients which do not use the request header. This will automatically align the UUID in the request headers and client telemetry.
...
Pre-KIP-1313 - broker does not expect ClientInstanceId in request header and ignores it
Client sends | Broker responds | Notes |
GetTelemetrySubscriptions v0 request.ClientInstanceId = 0 |
response |
This is KIP-714 initial GetTelemetrySubscriptions .
. |
The client will henceforth use UUID-B for client telemetry.
GetTelemetrySubscriptions v0
request.ClientInstanceId = UUID- |
B response.ErrorCode = NONE | This is KIP-714 |
initial Client is requesting a new client instance ID from the broker. The client |
will henceforth use UUID- |
B for client telemetry. |
...
Client sends
Broker responds
Notes
GetTelemetrySubscriptions v0 |
request.ClientInstanceId = |
UUID-R | response.ClientInstanceId = |
0 response.ErrorCode = NONE | This is KIP-714 non-initial |
. The client is using UUID-R for client telemetry. |
Post-KIP-1313 - broker is aware of optional ClientInstanceId in request header
Client sends | Broker responds | Notes |
GetTelemetrySubscriptions v0 header. |
clientInstanceId not present or 0 request.ClientInstanceId = 0 | response.ClientInstanceId = UUID- |
B response.ErrorCode = NONE | This is KIP-714 initial |
pre-KIP-1313 client. The client is requesting a new client instance ID from the broker |
. The client will henceforth use UUID-B for client telemetry. | ||
GetTelemetrySubscriptions v0 header.ClientInstanceId not present or 0 request.ClientInstanceId = UUID-R | response.ClientInstanceId = 0 response.ErrorCode = NONE | This is KIP-714 non-initial The client is using UUID-R |
for client telemetry. |
GetTelemetrySubscriptions v0 header.ClientInstanceId |
= UUID-H request.ClientInstanceId = UUID- |
H | response.ClientInstanceId = 0 response.ErrorCode = NONE | This is KIP- |
714 |
post-KIP-1313 client. The client is using UUID- |
H for request headers and client telemetry. This is what we expect from the Apache Kafka Java client after this KIP. |
GetTelemetrySubscriptions v0 header.ClientInstanceId = UUID-H request.ClientInstanceId = |
0 | response.ErrorCode = |
This is KIP-714 non-initial GetTelemetrySubscriptions request from a post-KIP-1313 client.
INVALID_REQUEST | This is not allowed. If a client sends UUID-H in the request header, the request client instance ID must also be UUID-H. After this KIP, the client generates the client instance ID and does not ask the broker to do so |
. |
GetTelemetrySubscriptions v0 header.ClientInstanceId = UUID-H request.ClientInstanceId = UUID |
-R≠UUID-H | response.ErrorCode = |
INVALID_REQUEST | This is |
GetTelemetrySubscriptions request from a post-KIP-1313 client.For some reason, the client is using UUID-H for request headers, but UUID-R for client telemetry. This is allowed, but not preferred. Included for completenessnot allowed. If a client sends UUID-H in the request header, the request client instance ID must also be UUID-H. |
In summary, for GetTelemetrySubscriptions v0, here are the combinations:
| Old broker | New broker | |
|---|---|---|
| Old client | Initial request:
Subsequent requests:
|
| Initial request:
|
Initial request:
request
Subsequent requests:
| |
| New client | Initial request:
|
|---|
|
Subsequent requests:
|
| Initial request:
|
|
Subsequent requests:
|
Public Interfaces
Client API Changes
...
A very small behavioral change is made in the broker handling of GetTelemetrySubscriptions v0. If the client sends a zero is made in the broker handling of GetTelemetrySubscriptions v0.
A post-KIP-1313 client will not send a zero ClientInstanceId in the request body to ask the broker to assign an ID.
If the client sends a zero ClientInstanceId in the request body, it must not send a ClientInstanceId in the request header.
Similarly, if the client sends a ClientInstanceId in the request body header and it also sends a non-zero ClientInstanceId in the request header, rather than generating a new UUID, the broker will respond with the ClientInstanceId from the request header. This naturally aligns the request header and client telemetry values for ClientInstanceId body, the values must be the same. It is not permitted to use one client instance ID for request headers and a different client instance ID for telemetry.
As a result, the description for the ClientInstanceId field in the request becomes "Unique id for this client instance. If client sends ClientInstanceId in the header, must equal that value. If not, must be set to 0 on the first request." .
Compatibility, Deprecation, and Migration Plan
...