...

• Group (epoch=23)
• A
• B
• Target Assignment (epoch=23)
• A - partitions=[foo-0]
• B - partitions=[foo-1]
• Member Assignment
• A - epoch=23, partitions=[foo-0]
• B - epoch=23, partitions=[foo-1]

Online Migration

We starts with a generic group.

• Generic Group (generation=22)
• A
• B
• C
• Assignment
• A - partitions=[foo-0, foo-1]
• B - partitions=[foo-3, foo-4]
• C - partitions=[foo-2, foo-5]

A leaves and rejoins with the new protocol enabled. The group is converted. The current generation becomes the group epoch. The target assignment and the member assignments are created based on the current assignment.

• Group (epoch=22)
• A (upgraded)
• B
• C
• Target Assignment (epoch=22)
• A - partitions=[foo-0, foo-1]
• B - partitions=[foo-3, foo-4]
• C - partitions=[foo-2, foo-5]
• Member Assignment
• A - epoch=22, partitions=[foo-0, foo-1]
• B - epoch=22, partitions=[foo-3, foo-4]
• C - epoch=22, partitions=[foo-2, foo-5]

A uses the new protocol. B and C still use the old protocol.

B leaves the group. The group coordinator removes him and bumps the group epoch.

• Group (epoch=23)
• A (upgraded)
• C
• Target Assignment (epoch=22)
• A - partitions=[foo-0, foo-1]
• B - partitions=[foo-3, foo-4]
• C - partitions=[foo-2, foo-5]
• Member Assignment
• A - epoch=22, partitions=[foo-0, foo-1]
• C - epoch=22, partitions=[foo-2, foo-5]

The group coordinator computes a new target assignment and installs it. It also triggers a rebalance for C. 

• Group (epoch=23)
• A (upgraded)
• C (PreparingRebalance)
• Target Assignment (epoch=23)
• A - partitions=[foo-0, foo-1, foo-3]
• C - partitions=[foo-2, foo-5, foo-4]
• Member Assignment
• A - epoch=22, partitions=[foo-0, foo-1]
• C - epoch=22, partitions=[foo-2, foo-5]

C heartbeats and is notified that a rebalance is required. C revokes all its partitions (assuming Eager protocol is used here) and sends a JoinGroup request.

The group coordinator sees that C does not own any partitions any more, so it can transition to epoch 23 and transition to CompletingRebalance.

• Group (epoch=23)
• A (upgraded)
• C (CompletingRebalance)
• Target Assignment (epoch=23)
• A - partitions=[foo-0, foo-1, foo-3]
• C - partitions=[foo-2, foo-5, foo-4]
• Member Assignment
• A - epoch=22, partitions=[foo-0, foo-1]
• C - epoch=23, partitions=[foo-2, foo-5, foo-4]

In the meantime, A heartbeats and transitions to epoch 23 as well.

• Group (epoch=23)
• A (upgraded)
• C (CompletingRebalance)
• Target Assignment (epoch=23)
• A - partitions=[foo-0, foo-1, foo-3]
• C - partitions=[foo-2, foo-5, foo-4]
• Member Assignment
• A - epoch=23, partitions=[foo-0, foo-1, foo-3]
• C - epoch=23, partitions=[foo-2, foo-5, foo-4]

C sends the SyncGroup request and collects his new assignment. All partitions are given because they are all free. C transitions to Stable.

• Group (epoch=23)
• A (upgraded)
• C (Stable)
• Target Assignment (epoch=23)
• A - partitions=[foo-0, foo-1, foo-3]
• C - partitions=[foo-2, foo-5, foo-4]
• Member Assignment
• A - epoch=23, partitions=[foo-0, foo-1, foo-3]
• C - epoch=23, partitions=[foo-2, foo-5, foo-4]

B rejoins the group with the new protocol. The group coordinator adds it and bumps the group epoch.

• Group (epoch=24)
• A (upgraded)
• B (upgraded)
• C (Stable)
• Target Assignment (epoch=23)
• A - partitions=[foo-0, foo-1, foo-3]
• C - partitions=[foo-2, foo-5, foo-4]
• Member Assignment
• A - epoch=23, partitions=[foo-0, foo-1, foo-3]
• A - epoch=0, partitions=[]
• C - epoch=23, partitions=[foo-2, foo-5, foo-4]

The group coordinator computes a new target assignment. A rebalance is triggered for C to revoke foo-4.

• Group (epoch=24)
• A (upgraded)
• B (upgraded)
• C (PreparingRebalance)
• Target Assignment (epoch=24)
• A - partitions=[foo-0, foo-1]
• B - partitions=[foo-3, foo-4]
• C - partitions=[foo-2, foo-5]
• Member Assignment
• A - epoch=23, partitions=[foo-0, foo-1, foo-3]
• B - epoch=0, partitions=[]
• C - epoch=23, partitions=[foo-2, foo-5, foo-4]

A heartbeats and he is told to revoke foo-3.

B heartbeats and transitions to epoch 24 but does not get any partitions yet because they are not free.

C heartbeats and he is told to rebalance. He revokes all its partitions and sends the JoinGroup request.

The group coordinator sees that C does not own any partitions any more, so it can transition to epoch 24 and transition to CompletingRebalance.

• Group (epoch=24)
• A (upgraded)
• B (upgraded)
• C (CompletingRebalance)
• Target Assignment (epoch=24)
• A - partitions=[foo-0, foo-1]
• B - partitions=[foo-3, foo-4]
• C - partitions=[foo-2, foo-5]
• Member Assignment
• A - epoch=23, partitions=[foo-0, foo-1, foo-3]
• B - epoch=24, partitions=[foo-3, foo-4]
• C - epoch=24, partitions=[foo-2, foo-5]

C sends the SyncGroup request to collect its assignment. He transitions to Stable.

• Group (epoch=24)
• A (upgraded)
• B (upgraded)
• C (Stable)
• Target Assignment (epoch=24)
• A - partitions=[foo-0, foo-1]
• B - partitions=[foo-3, foo-4]
• C - partitions=[foo-2, foo-5]
• Member Assignment
• A - epoch=23, partitions=[foo-0, foo-1, foo-3]
• B - epoch=24, partitions=[foo-3, foo-4]
• C - epoch=24, partitions=[foo-2, foo-5]

A heartbeats. He confirms the revocation of foo-3. He transitions to epoch 24.

• Group (epoch=24)
• A (upgraded)
• B (upgraded)
• C (Stable)
• Target Assignment (epoch=24)
• A - partitions=[foo-0, foo-1]
• B - partitions=[foo-3, foo-4]
• C - partitions=[foo-2, foo-5]
• Member Assignment
• A - epoch=24, partitions=[foo-0, foo-1]
• B - epoch=24, partitions=[foo-3, foo-4]
• C - epoch=24, partitions=[foo-2, foo-5]

B heartbeats and gets his assignment.

Compatibility, Deprecation, and Migration Plan

...