...

At the moment, the ListGroups API always returns all groups. There are metrics that track the number of groups in several states but the only way to know the state of a group is to describe it using the DescribeGroups API. In large clusters with hundreds or thousands of groups, it can be hard for a cluster administrator to identify which groups are stable or which groups are dead and can be deleted for example. Basically this requires describing a large number of groups which is inefficiency in terms of time for the administrator and load for the cluster. For example listing stable groups allows to immediately tell if several applications are healthy if their groups are all returned without needing to describe each group. This would allow building tools and user interfaces that can easily and quickly display informations about live groups.

Public Interfaces

ListGroups API, new version 3Update ListGroupsRequest to include a new flexible field "states".

ListGroups Request (Version: 3)
  state => STRING      <-- New field

ListGroups Response (Version: 3) => throttle_time_ms error_code [groups]   <-- Same as V2
  throttle_time_ms => INT32
  error_code => INT16
  groups => group_id protocol_type 
    group_id => STRING
    protocol_type => STRING

...

{
  "apiKey": 16,
  "type": "request",
  "name": "ListGroupsRequest",
  // Version 1 and 2 are the same as version 0.
  //
  // Version 3 is the first flexible version.
  // KIP-518 adds the States flexible field
  "validVersions": "0-3",
  "flexibleVersions": "3+",
  "fields": [
    { "name": "States", "type": "[]string", "versions": "3+", "tag": 0, "taggedVersions": "3+",
      "about": "The states of the groups we want to list", "fields": [
      { "name": "Name", "type": "string", "versions": "3+", "about": "The group state" }
    ]}
  ]
}

At the moment, consumer group state is already serialized as a STRING in DescribeGroupsResponse. To save a few bytes we could use an ID and update ConsumerGroupState?, so serializing it the same way here. ListGroupsResponse is left untouched.

To enable this feature in the AdminClient API, ListConsumerGroupsOptions will be updated.

public class ListConsumerGroupsOptions extends AbstractOptions<ListConsumerGroupsOptions> {

    /**
     * Only groups in thisthese statestates will be returned by listConsumerGroups()
     * By default, all groups are returned
     */
    public void withStateinStates(StringList<ConsumerGroupState> statestates) {}

    public StringList<ConsumerGroupState> getStatestates() {}
}

Proposed Changes

• Update the ListGroups protocol definition and bump the version to 3
  
  • A new flexible field "statestates" is added to the request
  • The response stays the same
  • The broker will only return groups that are in the specified state. If no state is specified , the specified state is unknown, or version < 3, all groups are returned.

...

• Update the ConsumerGroupCommand tool also will be updated to support filtering by group state
  • A new command line argument "–state --state STATE" enables users to specify a state. This new option will only be valid when used with "--list".

...

• Allow specifying a filter in ListGroups to retrieve groups by name or by state using wildcard
  
  • Unclear if it's a common use case
  Allow specifying multiple states to retrieve groups in any of the provided state
  • The current response only contains the names, so if multiple states are provided, we don't know which state each group is in