...

First, the admin client uses the ConsumerGroupDescribe RPC which responds with error code GROUP_ID_NOT_FOUND (69)  and an empty error message. Next, the admin client falls back to the pre-KIP-848 DescribeGroups RPC in case it's a classic consumer group.  This RPC succeeds and responds with error code NONE (0)  and returns the group with a status of Dead . It looks like a dead consumer group. There is no option of an error message in the protocol in this case, because the RPC doesn't support it. Finally, this dead group is translated into the error message Error: Consumer group 'MYSHARE' does not exist . The output seems kind of acceptable, but the tool actually thinks it's dealing with a dead consumer group. It would be better if the ConsumerGroupDescribe RPC failed in a straightforward way.

...

This KIP introduces a command-line tool for displaying all of the groups and their types.

Finally, in In situations where command-line tools are used to administer a group of the wrong type, you’ll now be told the group type is wrong, rather than the group does not exist.

...

The new kafka-groups.sh tool makes all of this information available.

Describing groups using the admin client

The behavior of AdminClient.describeConsumerGroups(Collection<String>) seems a little unusual. You can describe a collection of group IDs, some of which might exist and others might not. Here's how the response is built:

...

In cases (1) and (2), the admin client considers the operation a success, and this means the KafkaFuture  for this group completes successfully. In cases (3) and (4), the admin client considers the operation unsuccessful, and this means the KafkaFuture for this group completes exceptionally.

The behavior of AdminClient.describeShareGroups(Collection<String>)  was modelled on this for consistency.

Case (2) is the tricky one because you can't readily tell what the dead group means. This is why the admin tools use output like Error: Consumer group 'MYSHARE" does not exist, even when the group ID is recognised and it's just the wrong type.

This gives a problem though because even though it'This gives a problem though because even though it's not difficult to determine which groups have the incorrect type, it would take a breaking change to the admin client to discover this situation if using the admin client.

As a result, this KIP introduces a new option on DescribeConsumerGroupOptions  called validateGroupType  which changes the behavior in the case where a group ID is the wrong group type. For case (2) above, the new INCONSISTENT_GROUP_ID_NOT_FOUNDTYPE  error in the ConsumerGroupDescribe response is translated into a GroupIdNotFoundException an InconsistentGroupTypeException containing the error message from the RPC response. Previously, the error was swallowed meaning there was no opportunity to obtain an error message from the broker, and that can be used by the admin tools.

Public Interfaces

Client API changes

...

This class is modified to extend org.apache.kafka.clients.admin.GroupListing .

...

DescribeConsumerGroupsOptions

The following methods are added to this class.

...

If validateGroupType  is set, when the ConsumerGroupDescribe RPC response contains the error code INCONSISTENT_GROUP_TYPE , the describe fails with InconsistentGroupTypeException . If it is not set, the error code is treated the same as GROUP_ID_NOT_FOUND  and the describe succeeds by returning a ConsumerGroupDescription  in Dead  state.

...

DescribeShareGroupsOptions

The following methods are added to this class.

/**
 * Set to true if describing a group which is not a share group fails.
 */
public DescribeConsumerGroupsOptionsDescribeShareGroupsOptions validateGroupType(boolean validateGroupType);

/**
 * Set to true if describing a group which is not a share group fails.
 */
public boolean validateGroupType();

...

The following new exception is added to the org.apache.kafka.common.errorerrors  package corresponding to the new error code in the Kafka protocol.

• InconsistentGroupTypeException  - Indicates that the group exists but the group type is inconsistent with the operation.

The error message in the RPCs gives more information about the failure.

Kafka protocol changes

This KIP adds the following error code to the Kafka protocol.

...

These RPCs are used by administrative tools and the new error code will help with the usability of the tools. Assuming that this KIP is delivered later than KIP-848, a new version of ConsumerGroupDescribe with no schema change will be required to support the new error code.

...

...

$ bin/kafka-groups.sh --bootstrap-server localhost:9092 --list --consumer
old-consumer-group
new-consumer-group
new-consumer-group
simple-consumer-group

To describe all of the consumer groups:

$ bin/kafka-groups.sh --bootstrap-server localhost:9092 --describe --consumer
GROUP                   TYPE          PROTOCOL
old-consumer-group      Classic       consumer
new-consumer-group      Consumer      consumer
simple-consumer-group   Classic

To describe all of the KIP-848 consumer groups:

$ bin/kafka-groups.sh --bootstrap-server localhost:9092 --describe --group-type consumer
GROUP                   TYPE          PROTOCOL
old-consumer-group      Classic       consumer
new-consumer-group      Consumer      consumer
simple-consumer-group   Classic

To describe all of the share groups:

...

• If the broker does not support ListGroups v5, setting a types filter using ListGroupOptions.withTypes() results in UnsupportedVersionException when the ListGroups request is serialized. (This is KIP-848 behavior.)
• If the broker receives a group type in TypesFilter  that it does not support in a ListGroups v5 (or later) request, the filter is treated as unknown group type which thus matches no groups. (This is KIP-848 behavior.)
• If the client receives a group type that it does not understand in a ListGroups v5 (or later) response, the group type is GroupType.UNKNOWN . This is because the group type in the ListGroups RPC response cannot be parsed into a value of the GroupType  enumeration, and the default value of UNKNOWN  is used in these situations.

As a result, there is no compatibility problem with ListGroups as a result of this KIP.

Test Plan

The feature will be thoroughly tested with unit and integration tests.

...