...

Since we are carrying over more information during rebalance, we should be alerted on the metadata size increase. So far the hard limit is 1MB per metadata response, which means if we carry over too much information, the new protocol could hit hard failure. This is a common pain point for finding better encoding scheme for metadata if we are promoting incremental rebalancing KIPs like 415 and 429. Some thoughts from Guozhang have been started in this JIRA and we will be planning to have a separate KIP discussing different encoding technologies and see which one could work.

Public Interfaces

We are going to add a new type of protocol called "streams" for the protocol type. 

We will be adding following new configs:

...

Compatibility, Deprecation, and Migration Plan

Only client side upgrade is required.  just need to change the encoded metadata so that the server could react with a rebalance based on:

• Metadata size increase
• No downtime upgrade due to change of protocolType
• Stream instance replacement
  • The right approach for a global application instance replacement is
    • Increase the capacity of the current stream job to 2
    • Mark existing stream instances as leaving
    • Learner tasks finished on new hosts, shutting down old ones.

...