Page History

...

At the moment, the ListGroups API always returns all groups. There are metrics that track the number of groups in several states but the only way to know the state of a group is to describe it using the DescribeGroups API. In large clusters with hundreds or thousands of groups, it can be hard for a cluster administrator to identify which groups are stable or which groups are dead and can be deleted for example. Basically this requires describing a large number of groups which is inefficiency inefficient in terms of time for the administrator and load for the cluster.

To improve that use case, I propose to allow listing groups by state. For example listing stable groups allows to immediately tell if several applications are healthy if their groups are all returned without needing to describe each group. This would allow building tools and user interfaces that can easily and quickly display informations about live groups.

...

{
  "apiKey": 16,
  "type": "request",
  "name": "ListGroupsRequest",
  // Version 1 and 2 are the same as version 0.
  //
  // Version 3 is the first flexible version.
  //
  // Version 4 adds the States flexible field (KIP-518).
  "validVersions": "0-4",
  "flexibleVersions": "3+",
  "fields": [
    { "name": "States", "type": "[]string", "versions": "34+", "tag": 0, "taggedVersions": "34+",
      "about": "The states of the groups we want to list", "fields": [
      { "name": "Name", "type": "string", "versions": "3+", "about": "The group state" }
    ]}
  ]
}

...

This KIP bumps the version of this request to avoid any version incompatibility issues with older broker. See rejected alternatives.

If States is not specified or if it's empty in the request , from version 4, all groups will be returned without their states, like in versions < 4. At the moment, consumer group state is already serialized as a STRING in DescribeGroupsResponse, so serializing it the same way here.

When attempting to list groups with states using an older broker, an UnsupportedVersionException will be raised in the client.

...

public class ListConsumerGroupsOptions extends AbstractOptions<ListConsumerGroupsOptions> {

    /**
     * Only groups in these states will be returned by listConsumerGroups()
     * If not set, all groups are returned without their states
     */
    public ListConsumerGroupsOptions inStates(Set<ConsumerGroupState> states) {}

    /**
     * All groups with their states will be returned by listConsumerGroups()
     */
    public ListConsumerGroupsOptions inAnyState() {}

    /**
     * Returns the list of States that are requested
     */
    public Set<ConsumerGroupState> states() {}
}

...

To expose this feature in the command line tools, ConsumerGroupCommand tool will be updated to support filtering by group states. A The existing flag "--state" will be updated to allow specifying a list of group states when used with "–list". When specified, the state of each group will be printed next to along the group id.

For example:

# Existing behaviour
./kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list
group1
group2

# Listing all groups with their states
./kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list --state
GROUP          STATE
group1         stable
group2         empty

# Listing groups in the stable state
./kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list --states stable
GROUP          STATE
group1         stable

...

• Allow specifying a filter in ListGroups to retrieve groups by name or by state using wildcard
  
  • Unclear if it's a common use case
• Add flexible fields without bumping the ListGroups versionKeep ListGroups v3
  • Keeping v3 would allow cases where brokers could support v3 but not the new optional field. In that case, if a client specifies States states in its request, brokers would still return the full list of groups without states. This looked confusing. In case broker don't support a feature, a UnsupportedVersionException is preferred.

...