...

In KIP-714, the client instance ID is created by the broker which responds to a client's first GetTelemetrySubscription  RPC. In KIP-848, initially the member ID was created by the broker, but subsequently KIP-1082 changed this so that the client creates its own member ID. As a result, this KIP also changes the definition of the client instance ID so that the client creates its own UUID before it makes its initial connection to the cluster, and then it uses it for all future requests made to all brokers by that client instance. In addition, when the client makes its first GetTelemetrySubscription  request, it will also supply the client instance ID which it created. There is no need to change the behaviour of the cluster in handling client telemetry requests, it will just be the case that the Apache Kafka Java client after this KIP no longer supplies a zero client instance ID expecting the broker to calculate the ID. However, for migration purposes, the existing behaviour of KIP-714 is still supported.

Apart from its use in client telemetry, the addition of the client instance ID has no significance to the broker. It is being added to improve traceability and problem determination.

...

{
  "type": "header",
  "name": "RequestHeader",
  // Version 0 was removed in Apache Kafka 4.0, Version 1 is the new baseline.
  //
  // Version 0 of the RequestHeader is only used by v0 of ControlledShutdownRequest.
  //
  // Version 1 is the first version with ClientId.
  //
  // Version 2 is the first flexible version.
  // Tag 0 introduces client instance ID (KIP-1313).
  "validVersions": "1-2",
  "flexibleVersions": "2+",
  "fields": [
    { "name": "RequestApiKey", "type": "int16", "versions": "0+",
      "about": "The API key of this request." },
    { "name": "RequestApiVersion", "type": "int16", "versions": "0+",
      "about": "The API version of this request." },
    { "name": "CorrelationId", "type": "int32", "versions": "0+",
      "about": "The correlation ID of this request." },

    // The ClientId string must be serialized with the old-style two-byte length prefix.
    // The reason is that older brokers must be able to read the request header for any
    // ApiVersionsRequest, even if it is from a newer version.
    // Since the client is sending the ApiVersionsRequest in order to discover what
    // versions are supported, the client does not know the best version to use.
    { "name": "ClientId", "type": "string", "versions": "1+", "nullableVersions": "1+", "flexibleVersions": "none",
      "about": "The client ID string." },
    { "name": "ClientInstanceId", "type": "uuid", "versions": "2+", "taggedVersions": "2+", "tag":  0, "ignorable":  "true",
      "about": "Unique id for this client instance." }
  ]
}

GetTelemetrySubscriptions API

A new version 1 of this RPC is introduced which removes ClientInstanceId from the request and the response. Here's the updated request.

{
  "apiKey": 71,
  "type": "request",
  "listeners": ["broker"],
  "name": "GetTelemetrySubscriptionsRequest",
  "validVersions": "0-1",
  "flexibleVersions": "0+",
  "fields": [
    {
      "name": "ClientInstanceId", "type": "uuid", "versions": "0", <<<- not present in v1
      "about": "Unique id for this client instance, must be set to 0 on the first request."
    }
  ]
}

PushTelemetry API

A new version 1 of the RPC is introduced with removes ClientInstanceId  from the request. Here's the updated request:

...

After this KIP, the Apache Kafka Java client will send the ClientInstanceId  in the request header and also in the request body. If present in the header, it must match the value in the request body, and that value will not be zero.

It is still permitted to send the value zero for the ClientInstanceId  in the request, which will cause the broker to create a ClientInstanceId and send it back in the response. This is the original KIP-714 behavior and it is still supported for clients which do not use the ClientInstanceId  in the request header.

PushTelemetry API

After this KIP, the Apache Kafka Java client will send the ClientInstanceId  in the request header and also in the request body. If present in the header, it must match the value in the request body, and that value will not be zero.

Compatibility, Deprecation, and Migration Plan

...