Status

Current state: DiscussionRejected

Discussion thread: here 

JIRA: KAFKA-6583 

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

...

The class in which the rebalance protocol's metadata is contained is SubscriptionInfo which encodes information into a ByteBuffer. What this upgrade would require is to change the format of the encoding and decoding to the ByteBuffer in SubscriptionInfo, which could potentially break older versions of Kafka brokers due to incompatible metadata formatwould require us to give an upgrade strategy for older Kafka versions. A new encoding and decoding format will be introduced: mainly the addition of an extra integer for each TaskId in SubscriptionInfo. 

...

In this manner, when TaskId#encode() and TaskId#decode() is called in SubscriptionInfo, the number of state stores will also be processed.  Note that setNumberOfStateStores(...) will also be called during StreamTask instantion to set the number of state stores to the given ProcessorTopology's state store list size.

It would also be helpful to know that in the older version of encoding TaskId, only the topic group id and partition number was included. In this newer version, the number of state stores is also made available.

Compatibility, Deprecation, and Migration Plan

Since we are updating metadata, this will undoubtedly necessitate an upgrade path which has been provided in KIP-268.

Rejected Alternatives