...

The only addition is a numeric generation marker that increases during each improvement. This lets the leader detect the highest generation and ignore a consumer's reported assignment (if necessary) when that assignment belongs to older generations.

Generation Marker

There are two methods for implementing this generation marker:

1. A Sticky Assignor specific generation marker with the scope limited to Sticky Assignor (on the client side) only. This would mean that the generation marker of a sticky assignment may occasionally reset to 0 if no consumer in the group reports a previous assignment. But that does not jeopardize the logic presented above in resolving the scenarios that could lead to duplicate assignments. The upside of this method is the solution has limited scope and is specific to Sticky Assignor and other assignors or the AbstractPartitionAssignor class will not be impacted.
2. The consumer group's generation number can also be used for this purpose. The upside of this approach is that the assignment generation marker will be in-sync with consumer group's generation number. Also, the group's generation number, once available in the assignors, could later be used by other stateful partition assignors. The downside is added implementation complexity in exposing the consumer group's generation number to the Sticky Assignor which would potentially impact all existing assignor classes.  

One of these two methods will be used in the implementation.

Consumer group generation will be used for the new user data field. On top of addressing the cons described below in Rejected Alternatives section for a local generation marker, this option also makes it possible for other stateful assignors to use consumer group generation.

This generation is not currently exposed to any of partition assignors. It will be exposed as part of the assignor.assign(...) call in ConsumerCoordinator.java:

Compatibility, Deprecation, and Migration Plan

...

• Backward compatibility: If the leader uses the new protocol it first tries to deserialize reported assignments of consumers using the new protocol. If that does not work it falls back to using the old protocol assuming a default (e.g. -1) generation number for the assignment.
• Forward compatibility: If the leader uses the old protocol it deserializes only the array portion of the reported assignment byte array ignoring the potential integer that could be following. Generation markers reported by new clients will all be ignored in this case.

Rejected Alternatives

...

• Using a local generation marker within the scope of Sticky Assignor: This option was voted against because 1) it could cause confusion with the consumer group generation; 2) it could reset to 0 from time to time (if no consumer in the group has a previous assignment); 3) there are corner cases where it may not lead to the expected optimum assignment (example)