Status

Current state: Under Discussion

Discussion thread: here

JIRA: here

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

A Kafka broker has never had the ability to tell clients what version of Kafka it is. Today, at best, a client can use ApiVersions to determine the minimum and maximum supported version for all supported requests per broker. Since Kafka 0.8.0, every release, at least one request key was added or one request's version changed. Clients could use the response from ApiVersions to "guess" the version of Kafka they are talking to. So far for Kafka 3.4, no new request has been added and no existing request has bumped its version. If released today, clients will no longer be able to correctly guess which major/minor version of Kafka they are talking to. As well, for clients talking to other Kafka-like replacements (AWS Kinesis, Microsoft Event Hubs, Redpanda), it has been impossible to accurately guess the version because these systems support different sets of requests and request versions.

End users have found this version guessing very useful. For example, any UX management layer on top of Kafka that does not hook into Kafka metrics directly (AKHQ, Burrow, CMAK, Conduktor, Kafdrop, etc.), can use the version guess to display each broker’s version (or cluster version, which could be the minimum or maximum of all brokers) in the UX. Clients can also use the broker version in log lines so that end users can quickly see if a version looks about right or if something is seriously broken.

The version is useful even if the API surface area does not change. End users can quickly check from a client if the cluster is up to date, or if they may be exposed to bugs / problems from older broker versions. This is yet another area that users can ensure their systems are what they expect them to be.

This KIP is the broker-side complement of KIP-511.

Public Interfaces

ApiVersions API

Bump ApiVersionsRequest and ApiVersionsResponse to v4 and add two new required fields, BrokerSoftwareName and BrokerSoftwareVersion, to the response.

{
  "apiKey": 18,
  "type": "response",
  "name": "ApiVersionsResponse",
  // ...
  "validVersions": "0-4",
  "flexibleVersions": "3+",
  "fields": [
    { "name": "ErrorCode", "type": "int16", "versions": "0+",
      "about": "The top-level error code." },
    { "name": "ApiKeys", "type": "[]ApiVersion", "versions": "0+",
      "about": "The APIs supported by the broker.", "fields": [
      { "name": "ApiKey", "type": "int16", "versions": "0+", "mapKey": true,
        "about": "The API index." },
      { "name": "MinVersion", "type": "int16", "versions": "0+",
        "about": "The minimum supported version, inclusive." },
      { "name": "MaxVersion", "type": "int16", "versions": "0+",
        "about": "The maximum supported version, inclusive." }
    ]},
    { "name": "ThrottleTimeMs", "type": "int32", "versions": "1+", "ignorable": true,
      "about": "The duration in milliseconds for which the request was throttled due to a quota violation, or zero if the request did not violate any quota." },
    { "name":  "SupportedFeatures", "type": "[]SupportedFeatureKey", "ignorable": true,
      "versions":  "3+", "tag": 0, "taggedVersions": "3+",
      "about": "Features supported by the broker.",
      "fields":  [
        { "name": "Name", "type": "string", "versions": "3+", "mapKey": true,
          "about": "The name of the feature." },
        { "name": "MinVersion", "type": "int16", "versions": "3+",
          "about": "The minimum supported version for the feature." },
        { "name": "MaxVersion", "type": "int16", "versions": "3+",
          "about": "The maximum supported version for the feature." }
      ]
    },
    { "name": "FinalizedFeaturesEpoch", "type": "int64", "versions": "3+",
      "tag": 1, "taggedVersions": "3+", "default": "-1", "ignorable": true,
      "about": "The monotonically increasing epoch for the finalized features information. Valid values are >= 0. A value of -1 is special and represents unknown epoch."},
    { "name":  "FinalizedFeatures", "type": "[]FinalizedFeatureKey", "ignorable": true,
      "versions":  "3+", "tag": 2, "taggedVersions": "3+",
      "about": "List of cluster-wide finalized features. The information is valid only if FinalizedFeaturesEpoch >= 0.",
      "fields":  [
        {"name": "Name", "type": "string", "versions":  "3+", "mapKey": true,
          "about": "The name of the feature."},
        {"name":  "MaxVersionLevel", "type": "int16", "versions":  "3+",
          "about": "The cluster-wide finalized max version level for the feature."},
        {"name":  "MinVersionLevel", "type": "int16", "versions":  "3+",
          "about": "The cluster-wide finalized min version level for the feature."}
      ]
    },
    // -------------- START NEW FIELD --------------
    { "name": "BrokerSoftwareName", "type": "nullable-string", "versions": "4+", "ignorable": true,
      "about": "The name of the broker." },
    { "name": "BrokerSoftwareVersion", "type": "nullable-string", "versions": "4+", "ignorable": true,
      "about": "The version of the broker." }
    // -------------- END NEW FIELD --------------
  ]
}

Proposed Changes

This KIP proposes two new required fields in the ApiVersions response to permanently remove the need to "guess" a version going forward, and to add the ability to determine the name of the software the client is talking to. For Kafka, the idea is to return "Kafka" for the BrokerSoftwareName, and CURRENT_BROKER_VERSION for the BrokerSoftwareVersion. For other Kafka-like replacements, they can return something like "Kinesis" and ${kinesis_version}, or "Redpanda" and ${redpanda_version}.

Validation

Each system may name their software differently. Kafka, Kinesis, and Redpanda all use one word, while Event Hubs uses two. Optionally, Kinesis could be displayed as "Amazon Kinesis", and Event Hubs could be displayed as "Microsoft Event Hubs". This proposal does not prescribe any validation to the name field.

Each system may version their software differently. Kafka uses \d.\d, with the patch version largely being invisible to end users. Redpanda uses vYY.\d, with the patch version somewhat being invisible to end users (though it is relatively more important). It's unclear what versioning scheme Kinesis or Event Hubs uses, if any. Because each system may version uniquely, this proposal does not prescribe any validation to the version field.

Compatibility, Deprecation, and Migration Plan

N/A, only new clients will use these new fields. This does not change the behavior of existing clients.

Rejected Alternatives

Put BrokerSoftwareName and BrokerSoftwareVersion in the ResponseHeader

This alternative is mentioned to parallel KIP-511, but there is even less reason to include the broker name and version in every response. Clients are not interested in these pieces of information on every response, so we should not require it in every response. The ApiVersions response is the perfect place to put these fields: the client asks for ApiVersions likely once per connection, and in this single request, we can determine the text representation of the broker we are talking to.

Put BrokerSoftwareName and BrokerSoftwareVersion in the ResponseHeader but send it only once

Again paralleling KIP-511, these fields could be sent only in the first ResponseHeader to save bytes in subsequent requests. Lik KIP-511, it would be weird to have the information in random requests and it would make clients inconsistent.

Add a new request to communicate the broker metadata to the client

A new separate request/response could be used for the purpose. This is a valid option. This KIP considers the extra bit of information small enough, and the ApiVersions request infrequently enough, such that it's worth it to add these fields to ApiVersions.

Have BrokerSoftwareName and BrokerSoftwareVersion be tags

This would add no overhead if a broker does not want to tell the client its version. However, if a broker does not want to tell the client of its version, this defeats the purpose of the KIP. The broker should always tell the client of its version, thus there is no reason for these fields to be tags.

Have BrokerSoftwarename and BrokerSoftwareVersion be tags, and add an IncludeSoftwareNameVersion tag to ApiVersionsRequest

This solves the prior point: the response can be optional, but if the client asks, the broker should return the fields. Tag support is up to the broker, so it is possible that a client will ask and will not receive a response. As well, from a client implementation perspective, it is easier to just always ask for these fields, and it does not add much to the response. I worry that brokers will look like they support the full protocol, but then will not support this tag and it will be impossible for clients to determine what version of broker they are talking to.