Status

Current state: Under Discussion

Discussion thread: here

JIRA:  Unable to render Jira issues macro, execution error.

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

This KIP aims to solve two problems:

1. There is currently no way for a Kafka client to know which API version the broker supports. This means a client might not be able to perform its desired functionality, nor report any meaningful errors back to the application. Which leads in to the next problem, problem #2.

2. Unsupported requests are not handled gracefully by the broker. A broker receiving a protocol request it does not support will simply close the TCP connection to the client. From the client's point of view it has no way of knowing that the broker did not support the request, it could just as well be a network or software problem causing the connection reset.

This makes it hard for clients and applications to support multiple versions of Kafka, which in turn limits the Kafka eco-system since applications and clients will need to be manually built or configured for a specific broker version.

Additional key points

• The ProtocolVersionRequest may be used in broker-broker communication to decide on a common protocol version between brokers with different versions, i.e., during upgrades.

Public Interfaces

Add protocol ProtocolVersionRequest & Response message

ProtocolVersionRequest => (empty)
 
ProtocolVersionResponse => [ApiKey ApiName [ApiVersion]]
  ApiKey => int16                // API protocol key
  ApiName => string              // Human readable form of API, e.g. "ProduceRequest"
  ApiVersion => int16            // Supported versions in reverse order of preferredness, most preferred version last (thus typically ascending order)
  ApiDeprecatedVersion => int16  // Deprecated versions
 
Example in pseudo format:
 
ProtocolVersionResponse =>
[
 { ApiKey: 0, ApiName: "ProduceRequest", ApiVersion: [2,1,0], ApiDeprecatedVersion: [] },
 { ApiKey: 1, ApiName: "FetchRequest", ApiVersion: [1,0], ApiDeprecatedVersion: [] },
 { ApiKey: 2, ApiName: "OffsetRequest", ApiVersion: [2], ApiDeprecatedVersion: [0, 1] },
  ...
]

Proposed Changes

1. New protocol API: ProtocolVersionRequest and ProtocolVersionResponse

This KIP proposes a new protocol request and response to be added to the Kafka protocol, ProtocolVersionRequest and ProtocolVersionResponse. A client may use this new API to query the broker for its supported protocol request types and versions and use the information to select a suitable set of conforming API requests and versions for further communication with the broker.

If authentication is enabled, client must have permissions enabling "describe" action on "cluster" resource.

The ProtocolVersionResponse is only valid for the current connection to the broker; the client will need to reissue the ProtocolVersionRequest upon reconnect or when connecting to other brokers.

This API is optional for a client to implement.

2. Improved handling of unsupported requests on broker

Additionally, to solve issue nr 2. - ungraceful handling of unknown requests - it is also suggested to make the following change in the broker request handling code:

If a request is received for an unknown protocol request type, or for an unsupported version of a known type, the broker should respond with an empty response only including the common Length+CorrId header with Length=0, instead of closing the connection. The effect of this empty message received by the client is that the client will fail to parse the response (since it will expect a proper non-empty reply) and throw an error to the application with the correct context - that of a protocol parsing failure for request type XYZ. All existing clients should be able to handle this gracefully without any alterations to the code, while updated clients supporting the proposals in this KIP can handle the empty response accordingly by returning a "Request not supported by server" error the application. The level of detail in the returned error message to the application naturally varies between client implementations but is still by far better than a closed connection which says nothing of the underlying error.

Notes on managing the protocol specification

• The Kafka protocol specification will be moved to the kafka.git repository so it's versioned along with the code.
• The Kafka protocol specification of the latest Kafka release will be available on the homepage
• No request/response definitions are ever removed from the protocol specification. When removing support for a request type/version only the "Removed in: <brokerVersion>" information is added.
• Patches with additions to the Kafka protocol must also include corresponding updates to the protocol specification.

Compatibility, Deprecation, and Migration Plan

• Existing clients are not affected by 1. (ProtocolVersionRequest), but will get the error propagation benefits of 2. (graceful errors).
• Updated clients that add support for 1. will be able to adapt their functionality based on broker functionality and provide meaningful error propagation to applications when proper broker support is lacking.
• Release with 0.9.0.0: Since 0.9.0.0 will incorporate a large number of new protocol request types that the various client implementations are likely to pick up and implement it makes sense to add this new functionality along with it.

Rejected Alternatives

• Use an aggregated global ProtocolVersion
• Include meta key-value tags in response ("broker.id=...", "message.max.bytes=...", ...)
• Include user-defined tags in response ("aws.zone=...", "rack=...", ..)