Table of Contents |
---|
Status
Current state: DiscussionRejected
Discussion thread: here
JIRA: KAFKA-6583
...
The class in which the rebalance protocol's metadata is contained is SubscriptionInfo
which encodes information into a ByteBuffer
. What this upgrade would require is to change the format of the encoding and decoding to the ByteBuffer
in SubscriptionInfo
, which could potentially break older versions of Kafka brokers due to incompatible metadata formatwould require us to give an upgrade strategy for older Kafka versions. A new encoding and decoding format will be introduced: mainly the addition of an extra integer for each TaskId
in SubscriptionInfo
.
...
In this manner, when TaskId#encode()
and TaskId#decode()
is called in SubscriptionInfo
, the number of state stores will also be processed. Note that setNumberOfStateStores(...)
will also be called during StreamTask
instantion to set the number of state stores to the given ProcessorTopology
's state store list size.
It would also be helpful to know that in the older version of encoding TaskId
, only the topic group id and partition number was included. In this newer version, the number of state stores is also made available.
Compatibility, Deprecation, and Migration Plan
Since we are updating metadata, this will undoubtedly necessitate an upgrade path which has been provided in KIP-268.