You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 35 Next »

Status

Current state: Under Discussion

Discussion thread: here

JIRA:  Unable to render Jira issues macro, execution error.

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

This KIP aims to solve the problem that there is currently no way for a Kafka client to know which API version the broker supports. This means a client might not be able to perform its desired functionality, nor report any meaningful errors back to the application. This makes it hard for clients and applications to support multiple versions of Kafka, which in turn limits the Kafka eco-system since applications and clients will need to be manually built or configured for a specific broker version.

Proposed Changes

In order for a client to successfully talk to a broker, it needs to know what versions of various protocols does the broker support. The KIP suggests the following to achieve that.

  1. A new ApiVersionQueryRequest and Response type (version 0) will be added to allow clients to query the broker for supported API request types and versions.

    ApiVersionQueryRequest => ApiKeys
  ApiKeys = [INT16]
 
ApiVersionQueryResponse => ApiVersions
  ApiVersions = [ApiVersion]
    ApiVersion = ApiKey MinVersion MaxVersion
      ApiKey = INT16
      MinVersion = INT16
      MaxVersion = INT16
 

    1.  

  2. ApiVersionQueryRequest.ApiKeys semantics:

    1. Empty array (size 0): return all supported ApiVersions.

    2. Array of some ApiKeys: return ApiVersions for the requested keys.

  3. ApiVersionQueryResponse..MinVersion-MaxVersion semantics:
    1. MinVersion and MaxVersion dictates the lowest and highest supported API versions for the API key (inclusive).
    2. All versions in the MinVersion-MaxVersion must be supported by the broker. Specific version may be deprecated through protocol documentation but must still be supported (although it is fair to return an error code if the specific API supports it).

  4. Clients are recommended to use latest common supported API version.

  5. Deprecation of a protocol version will be done by marking a protocol version as deprecated in protocol documentation. Documentation shall also be used to indicate a protocol version that must not be used, or for any such information.For instance, say 0.9.0 had protocol versions [0] for api key 1. On trunk, version 1 of the api key was added. Users running off trunk started using version 1 of the api and found out a major bug. To rectify that version 2 of the api is added to trunk. For some reason, it is now deemed important to have version 2 of the api in 0.9.1 as well. To do so, version 1 and version 2 both of the api will be backported to the 0.9.1 branch. 0.9.1 broker will return 0 as min supported version for the api and 2 for the max supported version for the api. However, the version 1 should be clearly marked as deprecated on its documentation. It will be client's responsibility to make sure they are not using any such deprecated version to the best knowledge of the client at the time of development (or alternatively by configuration).

  6. Supported protocol versions obtained from a broker, is good only for current connection on which that information is obtained. In the event of disconnection, the client should obtain the information from the broker again, as the broker might have upgraded/ downgraded in the mean time.

  7. If a client faces connection closure on sending ApiVersionQueryRequest it could be due to following reasons.
    1. Broker does not support the ApiVersionQueryRequest API, the client should use some other method of determining the API versions to use (a safe default, configuration, error out, probing).
    2. Connection closure was caused by issues other than unknown API request.

The changes will be tested with librdkafka's implementation.

Public Interfaces

Addition of ApiVersionQueryRequest and Response version 0.

Compatibility, Deprecation, and Migration Plan

  • This is an optional interface and does not affect existing clients.
  • Updated clients that add support for the new ApiVersionQueryRequest will be able to adapt their functionality based on supported broker functionality.

Rejected Alternatives

  • Include meta key-value tags in response ("broker.id=...", "message.max.bytes=...", ...)
  • Include user-defined tags in response ("aws.zone=...", "rack=...", ..)
  • Using a versionInt instead of ApiVersion's id as broker protocol version.

  • Use a new protocol request and response, ProtocolVersionRequest and ProtocolVersionResponse. A client may use this new API to query the broker for its supported protocol request types and versions and use the information to select a suitable set of conforming API requests and versions for further communication with the broker.

  • Return array of supported versions, instead of min and max of supported versions, in MetadataResponse.
  • Return API versions in a new version of the MetadataRequest/Response

 

  • No labels