Status

Current state: Under Discussion Adopted

Discussion thread: here

JIRA:

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

...

1. The output of --describe will change to return one row per topic partition in the group. This means that group members with no assigned partition will not be present in the default output.
2. A new switch --members will be introduced, that can only be provided along with --describe, to return a list of all active members of the group. For each member the number of assigned topic partitions (without any offset information) and the assignment strategy used by the member are also returned. Additional information (such as member subscription and assignment) can also be provided with a --members --verbose option.
3. A new switch --A new switch --offsets will be introduced, that can only be provided along with --describe, that returns the same default output of --describe. I.e., this will be the default switch that can be additionally used with --describe.
4. A new switch --state will be introduced, that can only be provided along with --describe, to return group-level status information that is returned from the DescribeGroups API call.A new switch --subscription will be introduced, that can only be provided along with --describe, to list the topics each member is subscribed to.

Proposed Changes

To describe the proposed changes let us revisit the example above. This KIP proposes to change the --describe option of the ConsumerGroupCommand according to the following:

...

This option of the command returns offset and consumer information for each topic partition in the group. This is how the output of the example above would look like:

$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group
Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).
TOPIC                          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG        CONSUMER-ID                                       HOST                           CLIENT-ID
my-topic                       0          16577           33154           16577      consumer1-81fe8a2a-0b29-4964-a1d4-4741582132bb    /127.0.0.1                     consumer1
my-topic                       1          16578           33155           16577      consumer1-81fe8a2a-0b29-4964-a1d4-4741582132bb    /127.0.0.1                     consumer1
my-topic                       2          16577           33155           16578      consumer2-db6f12f2-e1de-4bb7-93ec-6170fefa8830    /127.0.0.1                     consumer2
my-second-topic                0          33023           44543           11520      consumer7-c1d056fa-eabc-42ed-b557-c848f747ec49    /127.0.0.1                     consumer7
my-other-topic                 1          7901            8652            751        -                                                 -                              -
my-other-topic                 0          7902            8654            752        -                                                 -                              -

Note that consumers without any assigned partitions will no longer appear in the listing.

--describe --members and --describe --members --verbose

A new option --members is introduced that is allowed with --describe only and returns info about active members of the group along with their assigned topic partitions:

$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group --members
Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).
CONSUMER-ID                                       HOST                           CLIENT-ID              #PARTITIONS     ASSIGNMENT-STRATEGY
consumer1-81fe8a2a-0b29-4964-a1d4-4741582132bb    /127.0.0.1                     consumer1              2               range
consumer2
consumer2-db6f12f2-e1de-4bb7-93ec-6170fefa8830    /127.0.0.1                     consumer2              1               range, round-robin
consumer7-c1d056fa-eabc-42ed-b557-c848f747ec49    /127.0.0.1                     consumer7              1               range

–describe --offsets

There is an additional --verbose switch that can be used along with this option to include additional details about each group memberThe new --offsets option, that is allowed with --describe only, produces the same result as the --describe option itself. In other words, --offsets is the default sub-option of --describe:

$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group --offsetsmembers --verbose
Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).
TOPICCONSUMER-ID                            PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG     HOST   CONSUMER-ID                        CLIENT-ID              #PARTITIONS HOST    ASSIGNMENT
consumer1-81fe8a2a-0b29-4964-a1d4-4741582132bb    /127.0.0.1                   CLIENT-ID
my-topic  consumer1              2       0          16577    my-topic(0,1)
consumer2-db6f12f2-e1de-4bb7-93ec-6170fefa8830    /127.0.0.1        33154           16577  consumer2    consumer1-81fe8a2a-0b29-4964-a1d4-4741582132bb    /127.0.0.1      1               consumer1
my-topic(2)
consumer7-c1d056fa-eabc-42ed-b557-c848f747ec49    /127.0.0.1                   1  consumer7        16578       1    33155           my-second-topic(0)

–describe --offsets

The new --offsets option, that is allowed with --describe only, produces the same result as the --describe option itself. In other words, --offsets is the default sub-option of --describe:

$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group --offsets
Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).
TOPIC16577      consumer1-81fe8a2a-0b29-4964-a1d4-4741582132bb    /127.0.0.1                     consumer1
my-topic                       2          16577        PARTITION   33155CURRENT-OFFSET  LOG-END-OFFSET  LAG        CONSUMER-ID  16578      consumer2-db6f12f2-e1de-4bb7-93ec-6170fefa8830    /127.0.0.1                     consumer2
my-second-topic      HOST          0          33023       CLIENT-ID
my-topic    44543           11520      consumer7-c1d056fa-eabc-42ed-b557-c848f747ec49    /127.0.0.1          16577           consumer7
my-other-topic33154           16577      1consumer1-81fe8a2a-0b29-4964-a1d4-4741582132bb    /127.0.0.1      7901            8652   consumer1
my-topic         751        -      1          16578           33155           16577           - consumer1-81fe8a2a-0b29-4964-a1d4-4741582132bb    /127.0.0.1                         -consumer1
my-other-topic                 0      2      7902    16577        8654   33155         752  16578      consumer2-db6f12f2-e1de-4bb7-93ec-6170fefa8830    /127.0.0.1                     consumer2
my-second-topic                0        -  33023           44543           11520      consumer7-

...

c1d056fa-eabc-

...

42ed-

...

$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group --state
Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).
PROTOCOL:b557-c848f747ec49    /127.0.0.1                     consumer7
my-other-topic                range
PROTOCOL-TYPE: consumer
STATE: 1         Stable

--describe --subscription

The --subscription option is also a sub-option of --describe. It is used to extract subscription information of each member in the group. For each member, a comma-separated list of topics the member is subscribed to is returned:

$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group --subscription
Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).
CONSUMER-ID  7901            8652            751        -                                                 -                              -
my-other-topic                 0          7902            8654            752        -                  HOST                           CLIENT-ID    -          SUBSCRIPTION
consumer1-81fe8a2a-0b29-4964-a1d4-4741582132bb    /127.0.0.1                     consumer1              my-topic
consumer2-db6f12f2-e1de-4bb7-93ec-6170fefa8830    /127.0.0.1                     consumer2              my-topic
consumer7-c1d056fa-eabc-42ed-b557-c848f747ec49    /127.0.0.1       -

–describe --state

The --state option, that can also be used with --describe only, reports the group level information that comes back in the response of DescribeGroups API call along with the coordinator id of the group:

$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group --state
Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).
COORDINATOR (ID)       ASSIGNMENT-STRATEGY     STATE          #MEMBERS
localhost:9092 (0)     range              consumer7     Stable         my-second-topic

...

3 

In the above outputs, the width of the following columns will be dynamically determined (as opposed to the fixed widths that have been applied so far):

• TOPIC
• CONSUMER-ID
• HOST
• CLIENT-ID
• COORDINATOR (ID)

Compatibility, Deprecation, and Migration Plan

...