Status

Current state: Under Discussion Adopted

Discussion thread: here

JIRA:

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

...

$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group
Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).
TOPIC                          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG        CONSUMER-ID                                       HOST                           CLIENT-ID
my-topic                       0 1         16577 30857          33154 118320          8746316577      consumer2consumer1-5d3a62da81fe8a2a-abe10b29-427d4964-a9e9a1d4-4962c35f2dae4741582132bb    /127.0.0.1                     consumer2consumer1
my-topic                       1 2         16578 30857          33155 118320          8746316577      consumer3consumer1-ad71338d81fe8a2a-d3790b29-4cff4964-be2da1d4-1e3971f32f514741582132bb    /127.0.0.1                     consumer3consumer1
my-topic                       02          3135716577           33155 118320          8696316578      consumer1consumer2-5e864292db6f12f2-bef0e1de-45f14bb7-9b9693ec-ea56943609e76170fefa8830    /127.0.0.1                     consumer1consumer2
my-second-topic                0          33023    -       44543   -        11520       -               -          consumer4-e173f09d-c761-4f4e-95c7-6fb73bb8fbffconsumer7-c1d056fa-eabc-42ed-b557-c848f747ec49    /127.0.0.1                     consumer4consumer7
-                              -          -               -               -          consumer5consumer8-7b80e4286078de6a-f8ffaf63-43f34c7a-83608bad-afd1c8ba43eaf03cb3606075    /127.0.0.1                     consumer5consumer8
my-other-topic                 1          68957901            68958652            0  751        -                                                 -                              -
my-other-topic                 0          68947902            68948654            0752          -                                                 -                              -

The information included in the command output includes:

...

1. The output of --describe will change to return one row per topic partition in the group. This means that group members with no assigned partition will not be present in the default output.
2. A new switch --members will be introduced, that can only be provided along with --describe, to return a list of all active members of the group. For each member the number of assigned topic partitions (without any offset information) and the assignment strategy used by the member are also returned. Additional information (such as member subscription and assignment) can also be provided with a --members --verbose option.
3. A new switch --offsets will be introduced, that can only be provided along with --describe, that returns the same default output of --describe. I.e., this will be the default switch that can be additionally used with --describe.
4. A new switch --state will be introduced, that can only be provided along with --describe, to return group-level status information that is returned from the DescribeGroups API call.
5. A new switch --subscription will be introduced, that can only be provided along with --describe, to list the topics each member is subscribed to.

Proposed Changes

To describe the proposed changes let us revisit the example above. This KIP proposes to change the --describe option of the ConsumerGroupCommand according to the following:

...

This option of the command returns offset and consumer information for each topic partition in the group. This is how the output of the example above would look like:

$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group
Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).
TOPIC                          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG        CONSUMER-ID                                       HOST                           CLIENT-ID
my-topic                       10          3085716577           11832033154           8746316577      consumer2consumer1-5d3a62da81fe8a2a-abe10b29-427d4964-a9e9a1d4-4962c35f2dae4741582132bb    /127.0.0.1                     consumer2consumer1
my-topic                       1 2         16578 30857          33155 118320          8746316577      consumer3consumer1-ad71338d81fe8a2a-d3790b29-4cff4964-be2da1d4-1e3971f32f514741582132bb    /127.0.0.1                     consumer3consumer1
my-topic                       2 0         16577 31357          33155 118320          8696316578      consumer1consumer2-5e864292db6f12f2-bef0e1de-45f14bb7-9b9693ec-ea56943609e76170fefa8830    /127.0.0.1                     consumer1consumer2
my-othersecond-topic                0 1         33023 6895          44543  6895         11520   0   consumer7-c1d056fa-eabc-42ed-b557-c848f747ec49    /127.0.0.1   -                  consumer7
my-other-topic                 1          7901    -        8652              751        -
my-other-topic                 0          6894            6894          -  0          -                  -
my-other-topic                 0          7902            8654            752        -                              -

--describe --members

                   -                              -

Note that consumers without any assigned partitions will no longer appear in the listing.

--describe --members and --describe --members --verbose

A new option --members is introduced that is allowed with --describe only and returns info about active members of the group along with their assigned topic partitions:

$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group --members
Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).
CONSUMER-ID                                       HOST                           CLIENT-ID                   TOPIC(PARTITIONS)
consumer2-5d3a62da-abe1-427d-a9e9-4962c35f2dae#PARTITIONS
consumer1-81fe8a2a-0b29-4964-a1d4-4741582132bb    /127.0.0.1                     consumer2consumer1                   my-topic(1)
consumer3-ad71338d-d379-4cff-be2d-1e3971f32f512
consumer2-db6f12f2-e1de-4bb7-93ec-6170fefa8830    /127.0.0.1                     consumer3consumer2              1
consumer7-c1d056fa-eabc-42ed-b557-c848f747ec49     my-topic(2)
consumer1-5e864292-bef0-45f1-9b96-ea56943609e7    /127/127.0.0.1                     consumer1consumer7                   my-topic(0)

Note that the TOPIC(PARTITIONS) column contains a comma separate list of topics the member is consuming from and corresponding partitions of each topic appear in parentheses (also in comma separated format). For example: my-topic(0,2),my-other-topic(1,3)

–describe --partitions

A new option --partitions is introduced that is allowed with --describe only and for each topic partition the group has stored offset for it reports the current offset, log end offset, and the lag:

1

There is an additional --verbose switch that can be used along with this option to include additional details about each group member:

$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group --members --verbose
Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).
CONSUMER-ID                                       HOST                           CLIENT-ID              #PARTITIONS     ASSIGNMENT
consumer1-81fe8a2a-0b29-4964-a1d4-4741582132bb    /127.0.0.1                     consumer1              2               my-topic(0,1)
consumer2-db6f12f2-e1de-4bb7-93ec-6170fefa8830    /127.0.0.1                     consumer2              1               my-topic(2)
consumer7-c1d056fa-eabc-42ed-b557-c848f747ec49    /127.0.0.1                     consumer7              1               my-second-topic(0)

–describe --offsets

The new --offsets option, that is allowed with --describe only, produces the same result as the --describe option itself. In other words, --offsets is the default sub-option of --describe:

$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group --offsets
Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).
TOPIC                          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG        CONSUMER-ID                                       HOST                           CLIENT-ID
my-topic                       0          16577           33154           16577      consumer1-81fe8a2a-0b29-4964-a1d4-4741582132bb    /127.0.0.1                     consumer1
my-topic                       1          16578           33155           16577      consumer1-81fe8a2a-0b29-4964-a1d4-4741582132bb    /127.0.0.1                     consumer1
my-topic                       2          16577           33155           16578      consumer2-db6f12f2-e1de-4bb7-93ec-6170fefa8830    /127.0.0.1                     consumer2
my-second-topic                0          33023           44543           11520      consumer7-c1d056fa-eabc-42ed-b557-c848f747ec49    /127.0.0.1                     consumer7
my-other-topic                 1          7901            8652            751        -                     

$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group --partitions
Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).
TOPIC                          PARTITION  CURRENT-OFFSET     LOG-END-OFFSET  LAG
my-topic                       1-
my-other-topic          30857       0    118320      7902    87463
my-topic        8654            752   2     -     30857           118320          87463
my-topic                       0-          31357           118320          86963
my-other-topic                 1          6895 -

–describe --state

The --state option, that can also be used with --describe only, reports the group level information that comes back in the response of DescribeGroups API call along with the coordinator id of the group:

$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group --state
Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).
COORDINATOR (ID)       ASSIGNMENT-STRATEGY    6895 STATE          #MEMBERS
localhost:9092 0
my-other-topic(0)     range            0       Stable   6894            6894            03 

In the above outputs, the width of the following columns will be dynamically determined (as opposed to the fixed widths that have been applied so far):

• TOPIC
• CONSUMER-ID
• HOST
• CLIENT-ID
• COORDINATOR (ID)

Compatibility, Deprecation, and Migration Plan

...

Users who use the new-consumer based consumer groups and somehow rely on the output of the --describe option of the ConsumerGroupCommand may have to adjust their clients to use one of the new options provided: --describe, –describe --members, or –describe --partitions; for example, in case they rely on the fact the command currently returns members with no assignment. Hopefully, how they use the command output will be simplified with these new options. 

...