You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 30 Next »

Status

Current state: Under Discussion

Discussion thread: here

JIRA:  Unable to render Jira issues macro, execution error.

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

This KIP aims to solve the problem that there is currently no way for a Kafka client to know which API version the broker supports. This means a client might not be able to perform its desired functionality, nor report any meaningful errors back to the application. This makes it hard for clients and applications to support multiple versions of Kafka, which in turn limits the Kafka eco-system since applications and clients will need to be manually built or configured for a specific broker version.

Additional key points

  • The broker version from MetadataResponse may be used in broker-broker communication to decide on a common protocol version between brokers with different versions, i.e., during upgrades.

Proposed Changes

In order for a client to successfully talk to a broker, it needs to know what versions of various protocols does the broker support. The KIP suggests the following to achieve that.

  1. New Metadata Request and Response version, v1, will be added to provide information on supported protocol versions to clients.

    MetadataRequest => api_keys, topics
	api_keys => [INT16]
	topics => [STRING]
 
MetadataResponse => api_versions, brokers, topic_metadata
	api_versions => [single_api_versions]
		single_api_versions => api_key, min_version, max_version
			api_key => INT16
			min_version => INT16
			max_version => INT16
	brokers => node_id host port
    	node_id => INT32
    	host => STRING
    	port => INT32
  	topic_metadata => topic_error_code topic [partition_metadata] 
    	topic_error_code => INT16
    	topic => STRING
    	partition_metadata => partition_error_code partition_id leader [replicas] [isr] 
      		partition_error_code => INT16
      		partition_id => INT32
      		leader => INT32

    Metadata Request v1 can be used by clients to ask for supported protocol versions by a broker. Client can send Metadata Request, v1, with api_keys set to one of the following values.

    1. Empty array, this will get the client supported versions for all ApiKeys.

    2. Array of some ApiKeys, this will get client supported versions for the specified ApiKeys.

    3. Null array, this will not get any information on ApiKeys. 

  2. Broker on receiving a Metadata Request v1, will respond with MetadataResponse v1. The response will contain information on supported versions for api keys specified in the request, if an empty array of api keys is sent, then broker will respond with info on all api keys, and if null array is sent for api keys, then no information on protocol versions is added to the response. Note that broker only provides information on its supported protocol versions.

  3. Clients can choose to not receive topics metadata by specifying a null array for topics in Metadata Request v1.

  4. Deprecation of a protocol version will be done by marking a protocol version as deprecated in protocol documentation. Documentation shall also be used to indicate a protocol version that must not be used, or for any such information.For instance, say 0.9.0 had protocol versions [0] for api key 1. On trunk, version 1 of the api key was added. Users running off trunk started using version 1 of the api and found out a major bug. To rectify that version 2 of the api is added to trunk. For some reason, it is now deemed important to have version 2 of the api in 0.9.1 as well. To do so, version 1 and version 2 both of the api will be backported to the 0.9.1 branch. 0.9.1 broker will return 0 as min supported version for the api and 2 for the max supported version for the api. However, the version 1 should be clearly marked as deprecated on its documentation. It will be client's responsibility to make sure they are not using any such deprecated version.

  5. Supported protocol versions obtained from a broker, is good only for current connection on which that information is obtained. In the event of disconnection, the client should obtain the information from the broker again, as the broker might have upgraded/ downgraded in the mean time.

  6. If a client faces connection closure on sending MetadataRequest Vn, where n is >=1, seeking supported protocol versions, it could be due to following reasons.
    1. Broker does not support version Vn of MetadataRequest. It is advised that client starts from V1 of metadata request and incrementally try above versions for connection closures.
    2. Connection closure was caused by issues other than unknown MetadataRequest version.

Public Interfaces

Changes to MetadataRequest and MetadataResponse

MetadataRequest and MetadataResponse v1 will be added.

Compatibility, Deprecation, and Migration Plan

  • Updated clients that add support for new MetadataRequest will be able to adapt their functionality based on broker functionality.

Rejected Alternatives

  • Include meta key-value tags in response ("broker.id=...", "message.max.bytes=...", ...)
  • Include user-defined tags in response ("aws.zone=...", "rack=...", ..)
  • Using a versionInt instead of ApiVersion's id as broker protocol version.

  • Use a new protocol request and response, ProtocolVersionRequest and ProtocolVersionResponse. A client may use this new API to query the broker for its supported protocol request types and versions and use the information to select a suitable set of conforming API requests and versions for further communication with the broker.

  • Return array of supported versions, instead of min and max of supported versions, in MetadataResponse.

 

  • No labels