Status

Current state: Under Discussion

Discussion thread: here

JIRA: KAFKA-5526

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

As far as describing a consumer group goes, the ConsumerGroupCommand currently lists all members of the group along with associated topic partitions and their committed offsets all in one view. For example:

$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group
Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).
TOPIC                          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG        CONSUMER-ID                                       HOST                           CLIENT-ID
my-topic                       1          30857           118320          87463      consumer2-5d3a62da-abe1-427d-a9e9-4962c35f2dae    /127.0.0.1                     consumer2
my-topic                       2          30857           118320          87463      consumer3-ad71338d-d379-4cff-be2d-1e3971f32f51    /127.0.0.1                     consumer3
my-topic                       0          31357           118320          86963      consumer1-5e864292-bef0-45f1-9b96-ea56943609e7    /127.0.0.1                     consumer1
-                              -          -               -               -          consumer4-e173f09d-c761-4f4e-95c7-6fb73bb8fbff    /127.0.0.1                     consumer4
-                              -          -               -               -          consumer5-7b80e428-f8ff-43f3-8360-afd1c8ba43ea    /127.0.0.1                     consumer5
my-other-topic                 1          6895            6895            0          -                                                 -                              -
my-other-topic                 0          6894            6894            0          -                                                 -                              -

The information included in the command output includes:

• all topic partitions the group is consuming or has consumed from (including those without an active consumer)
• current offset, log end offset, and the lag for each topic partition in the group
• all active members of the group (including those without an assigned topic partition)

This KIP suggests breaking down the command to cover some of the information pieces above separately using additional switches. There are a couple of reasons behind this improvement:

• The above output, with all the missing column values, looks weird and could be difficult to interpret. For a big group, where there are many consumers without assigned partition, the command will output a long list, while the user may only be interested in, for example, active members only.
• A user is typically interested in one of the information pieces above when issuing the command: either offset information, or members, or topic partitions associated with the group.

As part of this KIP, additional information about the group will also be generated that is currently missing. For example, generation id of the group, or the assignment policy used by each member. 

Public Interfaces

The change required for this includes updating the ConsumerGroupCommand tool:

1. The output of --describe will change to return one row per topic partition in the group. This means that group members with no assigned partition will not be present in the default output.
2. A new switch --members will be introduced, that can only be provided along with --describe, to return a list of all active members of the group. For each member the number of assigned topic partitions (without any offset information) and the assignment strategy used by the member are also returned.
3. A new switch --offsets will be introduced, that can only be provided along with --describe, that returns the same default output of --describe. I.e., this will be the default switch that can be additionally used with --describe.
4. A new switch --state will be introduced, that can only be provided along with --describe, to return group-level status information that is returned from the DescribeGroups API call.
5. A new switch --subscription will be introduced, that can only be provided along with --describe, to list the topics each member is subscribed to.

Proposed Changes

To describe the proposed changes let us revisit the example above. This KIP proposes to change the --describe option of the ConsumerGroupCommand according to the following:

--describe

This option of the command returns information on only active members of the group with assigned partitions. This is how the output of the example above would look like:

$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group
Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).
TOPIC                          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG        CONSUMER-ID                                       HOST                           CLIENT-ID
my-topic                       1          30857           118320          87463      consumer2-5d3a62da-abe1-427d-a9e9-4962c35f2dae    /127.0.0.1                     consumer2
my-topic                       2          30857           118320          87463      consumer3-ad71338d-d379-4cff-be2d-1e3971f32f51    /127.0.0.1                     consumer3
my-topic                       0          31357           118320          86963      consumer1-5e864292-bef0-45f1-9b96-ea56943609e7    /127.0.0.1                     consumer1

--describe --members

A new option --members is introduced that is allowed with --describe only and returns info about active members of the group along with their assigned topic partitions:

$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group --members
Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).
CONSUMER-ID                                       HOST                           CLIENT-ID                   TOPIC(PARTITIONS)
consumer2-5d3a62da-abe1-427d-a9e9-4962c35f2dae    /127.0.0.1                     consumer2                   my-topic(1)
consumer3-ad71338d-d379-4cff-be2d-1e3971f32f51    /127.0.0.1                     consumer3                   my-topic(2)
consumer1-5e864292-bef0-45f1-9b96-ea56943609e7    /127.0.0.1                     consumer1                   my-topic(0)

Note that the TOPIC(PARTITIONS) column contains a comma separate list of topics the member is consuming from and corresponding partitions of each topic appear in parentheses (also in comma separated format). For example: my-topic(0,2),my-other-topic(1,3)

–describe --partitions

A new option --partitions is introduced that is allowed with --describe only and for each topic partition the group has stored offset for it reports the current offset, log end offset, and the lag:

$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group --partitions
Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).
TOPIC                          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG
my-topic                       1          30857           118320          87463
my-topic                       2          30857           118320          87463
my-topic                       0          31357           118320          86963
my-other-topic                 1          6895            6895            0
my-other-topic                 0          6894            6894            0

Compatibility, Deprecation, and Migration Plan

The proposed changes apply to the new Java-based consumer only. Therefore, the consumer groups based on the old consumer will be unaffected.

Users who use the new-consumer based consumer groups and somehow rely on the output of the --describe option of the ConsumerGroupCommand may have to adjust their clients to use one of the new options provided: --describe, –describe --members, or –describe --partitions. Hopefully, how they use the command output will be simplified with these new options. 

Rejected Alternatives