...

In the current architecture, during each rebalance consumer groups on broker side will assign new member id with a UUID randomly generated each time. This is to make sure we have unique identity for each group member. During client restart, consumer will send a JoinGroupRequest with a special UNKNOWN_MEMBER id, which has no intention to be treated as an existing member.  To make the KIP work, we need to change both client side and server side logic to make sure we persist member identity throughout restarts, which means we could reduce number of rebalances since we are able to apply the same assignment based on member identities. The idea is summarized as `static membership` static membership, which in contrary to dynamic membership (the one our system currently uses), is prioritizing "state persistence" over "liveness". Since for many stateful consumer/stream applications, the state shuffling is more painful than short time partial unavailability.

Proposed Changes

We will be introducing a bunch of new terms:

• Static Membership: the membership protocol where the consumer group will not trigger rebalance unless 1. a new member joins 2. a leader rejoins. 3. an existing member go offline over certain timeout.
• Member name: the unique identifier defined by user to distinguish each client instance.
• Member registration timeout: the max time we could tolerate a static member to go offline.
• Member expansion timeout: the max time we will wait since we receive a new static member join request.

...

There are cases when we are scaling down the application, it is advised to do it quickly so that when the registration timeout is reached since the first gone member, we could trigger one single rebalance and make the progress back on track. Note that here we are sacrificing liveness for 15 min of registration timeout for the sake of minimizing state shuffling.

Another case is adding new static memberships (scale up!). This operation should be happening fast enough (to make sure capacity could catch up quickly), we are defining another config called expansion timeout.

...

In this example unfortunately, we triggered two rebalances, because C is too late to catch first round expansion timeout. When C finally joins, it triggers the counter of expansion timeout. After 5 min, another rebalance kicks in and assign new tasks to C. 

Switch between static and dynamic membership

For the very first version, we hope to make the logic simple without confusing peopleenough. As we have mentioned, the goal of static membership is to reduce multiple rebalances with intermittent failures. There should be an easy way to fallback to dynamic membership when user prefers to let broker handle task assignments, for example there is no handy tool like K8 or other scheduling frameworkliveness becomes a more important factor when the state tuning has big progress. The fallback is simple: when the membership becomes stable, the first join group request will decide the membership protocol for next round. For example, when we are running stable with static membership, deploy a new change to the client app without member.name being set could invalidate the current hashmap on broker, and all the v4 request containing member.name will be treated as a normal dynamic member and perform the sticky assignment as usual. While we keep adding the static member into the group, the behavior will not change, when we shall wait until member.expansion.timeout to perform the rebalance. 

...

Compatibility, Deprecation, and Migration Plan

• This new change is an effort of reducing rebalances during consumer rolling restart. Since we introduced a new version of join group request, this should be a backward compatible changeThe fallback logic has been discussed previously.

Non-goal

We do have some offline discussions on handling leader rejoin case, for example since the broker could also do the subscription monitoring work, we don't actually need to trigger rebalance on leader side blindly based on its rejoin request. However this is a separate topic and we will address it in another KIP. 

...