Status

Current state: Under DiscussionImplemented

Discussion thread: Here(not yet set)

JIRA: KAFKA-4462

Motivation

...

In some cases, older brokers will be unable to perform a feature supported by newer clients.  For example, retrieving message offsets from a message timestamp (KIP-79) is not supported in versions before 0.10.1.  In these cases, the new client will throw an ObsoleteBrokerVersion exception when attempting to perform the operation.  This will let clients know that the reason why the operation cannot be performed is because the broker version in use does not support it.  At the same time, clients which do not attempt to use the new operations will proceed as before.

Implementation

 Each RPC has a version number which increases monotonically. There are currently about 20 RPCs, with names like Produce, Fetch, JoinGroup, SyncGroup, etc.  Currently, the server checks the RPC version of each RPC the client sends, to make sure it can process it.  This information is sent over the wire in the beginning of each RPC.  The server can handle several versions of client RPCs.  But the client always sends the newest version of each RPC that it knows about.

...

No public RPCs will be changed or created.  No new public methods will be added to clients.  A new public exception, ObsoleteVersionException, will be created.

There will be a new command which allows system administrators to find out what RPCs versions all brokers in the cluster support.  It will make an ApiVersionRequest to each broker, and then print out the results.  This may be useful for debugging and troubleshooting version compatibility issues.

The new command will be called BrokerVersionCommand.  It will print out broker versions on the command line.  The format will show the version range for each broker, as follows:

cmccabe@aurora:~/src/kafka> ./bin/kafka-broker-versions.sh --bootstrap-server localhost:9092 --listApiVersions
aurora:9092 (id: 0 rack: null) -> {
 Produce(0): 0 to 2,
 Fetch(1): 0 to 3,
 Offsets(2): 0 to 1,
 Metadata(3): 0 to 2,
 LeaderAndIsr(4): 0,
 StopReplica(5): 0,
 UpdateMetadata(6): 0 to 2,
 ControlledShutdown(7): 1,
 OffsetCommit(8): 0 to 2,
 OffsetFetch(9): 0 to 1,
 GroupCoordinator(10): 0,
 JoinGroup(11): 0 to 1,
 Heartbeat(12): 0,
 LeaveGroup(13): 0,
 SyncGroup(14): 0,
 DescribeGroups(15): 0,
 ListGroups(16): 0,
 SaslHandshake(17): 0,
 ApiVersions(18): 0,
 CreateTopics(19): 0,
 DeleteTopics(20): 0
}

Migration Plan and Compatibility

...

In the future, we might consider adding an API whereby clients can specify which features that they needed when connecting to the broker.  This would allow clients to know which features they were getting up-front, rather than learning later.  For example, a client could specify that it needs ListOffsetRequestV1 support in order to start up.  Or the client could specify that it needs exactly-once semantics in order to start up.

 If we ever need to update to a new, incomptible version of the SaslHandshakeRequest, we could use this mechanism to do so.  The client would simply make the ApiVersionsRequest call before setting up SASL.  This will probably not be necessary, since the current SaslHandshakeRequest is fairly flexible.  However, it is an option in the future if we need it.