THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
Status
Current state: "Under Discussion"
Discussion thread: here
JIRA: KAFKA-12473
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation
Now that 3.0 is coming up, we can change the default ConsumerPartitionAssignor to something better than the RangeAssignor. The original plan was to switch over to the StickyAssignor, but now that we have incremental cooperative rebalancing, we should consider using the new "CooperativeStickyAssignor, RangeAssignor" instead: this will enable the consumer group to follow the COOPERATIVE protocol, improving the rebalancing experience OOTB, if all consumers upgraded to the new byte-code. Once KAFKA-12477 is completed, we can ensure that all consumers will use CooperativeStickyAssignor after they all upgraded, or fall back to RangeAssignor if some consumers are in old version.
In Kafka, we currently support the following assignors:
| Supported strategy | Feature | Drawback | |
| RangeAssignor | Eager | Current default value. The 1st assignor we have | possible to generate heavily skewed assignments when the consumer topic subscriptions are not identical |
| RoundRobinAssignor | Eager | Improvement for RangeAssignor to have balanced assignment | Didn't consider the overheads of reassignment |
| StickyAssignor | Eager | Improvement for RoundRobinAssignor/RangeAssignor to preserve the existing assignments to reduce some of the overheads of a reassignment | Will have stop-the-world issue when doing rebalance |
| CooperativeStickyAssignor | Eager, Cooperative | To be default value in 3.0 as in this KIP described, by having multiple rounds of rebalance to avoid the stop-the-world issue as described in KIP-429 |
As above table showed, since we already introduced a better assignor strategy, we should change the default assignor now.
Public Interfaces
With this KIP, the default value of partition.assignment.strategy changes from "RangeAssignor" to "CooperativeStickyAssignor, RangeAssignor"
- It won't affect to the current consumers if current consumers have explicitly set the
partition.assignment.strategyvalue - If the consumers rely on the default value of
partition.assignment.strategy, users can do 1 of the following options:Set the
partition.assignment.strategyconfig to one of the non-cooperative assignors during the rolling upgrade if they wish to remain on EAGER and/or perform the upgrade in just a single rolling bounce.- Upgrading the consumer (refer to the following upgrade path section)
Proposed Changes
- Change the default setting of
partition.assignment.strategyin implementation and tests - When changing the default assignor we need to make sure this is clearly documented in the upgrade guide for 3.0
- After this KIP, the client will have:
New applications will enable cooperative rebalancing by default
Existing applications which don’t set an assignor can safely upgrade using a single rolling bounce with no extra steps, and will automatically transition to cooperative rebalancing
Existing applications which do set an assignor that uses EAGER can likewise upgrade their applications to COOPERATIVE with a single rolling bounce
Once on new version (ex: V3.0), applications can safely go back and forth between EAGER and COOPERATIVE
Applications can safely downgrade
- This change will also propagate to Connect, when consumer groups we bring up for sink tasks
Compatibility, Upgrade path
No special upgrade path is necessary.
Note: After KAFKA-12477 is introduced, we'll have the upgrade experience improvement to reduce the safe upgrade path to just a single rolling bounce. We also make sure there's "short circuit" when there's mixing the cooperative and eager rebalancing protocols and the group leader is in old byte code. Detailed implementation can be found in KAFKA-12477.
Rejected Alternatives
If there are alternative ways of accomplishing the same thing, what were they? The purpose of this section is to motivate why the design is the way it is and not some other way.