Status
Current state: Under Discussion
Discussion thread: [Discuss] KIP-571: Add option to force remove members in StreamsResetter
JIRA: KAFKA-9146
Motivation
Sometimes people want to reset the stream application sooner, but blocked by the left-over members inside group coordinator, which only expire after session timeout. When user configures a really long session timeout, it could prevent the group from clearing. We should consider adding the support to cleanup members by forcing them to leave the group. To do that,
- If the stream application is already on static membership, we could call directly from adminClient.removeMembersFromGroup
If the application is on dynamic membership, we should modify adminClient.removeMembersFromGroup interface to allow deletion based on member.id.
Public Interfaces
Client side changes:
public class MemberToRemove { private String memberId; public MemberToRemove() {} public MemberToRemove setGroupInstanceId(String groupInstanceId) {} public MemberToRemove setMemberId(String memberId) {} }
CmdLine API change:
forceOption = optionParser.accepts("force","Force remove members when long session time out has been configured, please make sure to shutdown all stream applications when this option is specified to avoid unexpected rebalances")
Proposed Changes
KIP-571 plan to support the ability to force delete members in StreamsResetter, this basically involves 2 public interfaces changes as below:
1) Changes in org.apache.kafka.clients.admin.MemberToRemove
Previously the org.apache.kafka.clients.admin.KafkaAdminClient#removeMembersFromConsumerGroup only support to remove members by specifying the groupInstanceId, which means it only support removal of static members, so the memberId is added to support removing dynamic members . Since there are two String fields now, new helpers are added and the old constructor will be deprecated.
2) Add cmdline option --force to StreamsResetter
While with the new option, the StreamsResetter will force removing all active members fetched by KafkaAdminClient#describeConsumerGroups, and all the members info will be logged out if all removals succeed. Otherwise, exception will be thrown. Further more, users should make sure all the stream applications should be shutdown when run StreamsResetter with --force, otherwise it might trigger unexpected rebalance.
Server side needs no change, GroupCoordinator#handleLeaveGroup can handle both dynamic and static member removals since 2.3.0.
Compatibility, Deprecation, and Migration Plan
- Because no classes/method will be removed but only deprecated, this change will be backward compatible
- The new added cmdline option --force is also backward compatible because it's new feature, if not specified, the StreamsResetter's behaviour remains unchanged.
Requirement to use this feature:
Release version requirement | Expected error if requirement not satisfied | |
---|---|---|
broker side | >= 2.4 | UnsupportedVersionException will be thrown because batch removal feature was introduced since version 2.4 |
client side | update to this KIP | "force is not a recognized option" will be in the client side log |
Rejected Alternatives
None