...

Once a new target assignment is installed, each member will independently reconcile their current assignment with their new target assignment. Ultimately, each member will converge to their target epoch and assignment. The reconciliation process requires two three phases:

...

1. The group coordinator revokes the partitions which are no longer in the target assignment of the member.

...

1. It does so by providing the Current Partitions - Target Partitions in the heartbeat response until the member acknowledges the revocation in the heartbeat response. The group coordinator will give the rebalance timeout to the member for the revocation process to complete or kick it out from the group otherwise.
2. When the group coordinator receives the acknowledgement of the revocation, it updates the member current assignment to its target assignment (and target epoch) and durably persist it.
3. The group coordinator

...

1. assigns the new partitions to the member.

...

1. It does so by providing the Target Partitions to the member while ensuring that partitions which are not revoked by other members yet are removed from this set. In other words, new partitions are incrementally assigned to the member when they are revoked by the other members.

The rebalance timeout is provided by the member when it joins the group. It is basically the max poll interval configured on the client side. The timer starts ticking when the heartbeat response is sent out by the group coordinator.

...

Each member encodes the lag of its standby tasks in its metadata. We can not update the lag in every heartbeat request because that would constantly trigger reassignment in the group. Instead, when a) the task lag has been reduced within the acceptable.recovery.lag threshold or b) the task lag is consistently increasing for some time, the member should consider triggering a rebalance by sending its next heartbeat with the appropriate encoded reason and the updated task lags. 

Upgrade Path

TODO

Cluster Upgrade

• Deploy the new version of the software
• Roll the cluster
• Enable the new protocol
• Roll the cluster

Group Upgrade

Eager and Cooperative

• Eager revokes all partitions before re-joining the group.
• Cooperative does two rebalance rounds. One to revoke partitions. One to assign partitions.

JoinGroup API

• topics → subscribed topics ids
• owned partitions → owned partitions (id based)
• user data → assignor metadata

SyncGroup API

Heartbeat API

• When do we need to trigger a rebalance?

Public Interfaces

This section lists the changes impacting the public interfaces.

...

Rejected Alternatives

TODO

Future Work

...

• Connect
• Leader Election / Membership API