Table of Contents |
---|
Status
Current state: Under Discussion Adopted
Discussion thread: here
JIRA: Jira server ASF JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key KAFKA-5526
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
...
- The output of
--describe
will change to return one row per topic partition in the group. This means that group members with no assigned partition will not be present in the default output. - A new switch
--members
will be introduced, that can only be provided along with--describe
, to return a list of all active members of the group. For each member the number of assigned topic partitions (without any offset information) and the assignment strategy used by the member are also returned. Additional information (such as member subscription and assignment) can also be provided with a--members
--verbose
option. - A new switch
--
A new switch--offsets
will be introduced, that can only be provided along with--describe
, that returns the same default output of--describe
. I.e., this will be the default switch that can be additionally used with--describe
. - A new switch
--state
will be introduced, that can only be provided along with--describe
, to return group-level status information that is returned from theDescribeGroups
API call.A new switch--subscription
will be introduced, that can only be provided along with--describe
, to list the topics each member is subscribed to.
Proposed Changes
To describe the proposed changes let us revisit the example above. This KIP proposes to change the --describe
option of the ConsumerGroupCommand
according to the following:
...
This option of the command returns offset and consumer information for each topic partition in the group. This is how the output of the example above would look like:
Code Block | ||
---|---|---|
| ||
$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers). TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID my-topic 0 16577 33154 16577 consumer1-81fe8a2a-0b29-4964-a1d4-4741582132bb /127.0.0.1 consumer1 my-topic 1 16578 33155 16577 consumer1-81fe8a2a-0b29-4964-a1d4-4741582132bb /127.0.0.1 consumer1 my-topic 2 16577 33155 16578 consumer2-db6f12f2-e1de-4bb7-93ec-6170fefa8830 /127.0.0.1 consumer2 my-second-topic 0 33023 44543 11520 consumer7-c1d056fa-eabc-42ed-b557-c848f747ec49 /127.0.0.1 consumer7 my-other-topic 1 7901 8652 751 - - - my-other-topic 0 7902 8654 752 - - - |
Note that consumers without any assigned partitions will no longer appear in the listing.
--describe --members
and --describe
--members
--verbose
A new option --members
is introduced that is allowed with --describe
only and returns info about active members of the group along with their assigned topic partitions:
Code Block | ||
---|---|---|
| ||
$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group --members Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers). CONSUMER-ID HOST CLIENT-ID #PARTITIONS ASSIGNMENT-STRATEGY consumer1-81fe8a2a-0b29-4964-a1d4-4741582132bb /127.0.0.1 consumer1 2 range consumer2 consumer2-db6f12f2-e1de-4bb7-93ec-6170fefa8830 /127.0.0.1 consumer2 1 range, round-robin consumer7-c1d056fa-eabc-42ed-b557-c848f747ec49 /127.0.0.1 consumer7 1 range |
–describe --offsets
There is an additional --verbose
switch that can be used along with this option to include additional details about each group memberThe new --offsets
option, that is allowed with --describe
only, produces the same result as the --describe
option itself. In other words, --offsets
is the default sub-option of --describe
:
Code Block | ||
---|---|---|
| ||
$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group --offsetsmembers --verbose Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers). TOPICCONSUMER-ID PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG HOST CONSUMER-ID CLIENT-ID #PARTITIONS HOST ASSIGNMENT consumer1-81fe8a2a-0b29-4964-a1d4-4741582132bb /127.0.0.1 CLIENT-ID my-topic consumer1 2 0 16577 my-topic(0,1) consumer2-db6f12f2-e1de-4bb7-93ec-6170fefa8830 /127.0.0.1 33154 16577 consumer2 consumer1-81fe8a2a-0b29-4964-a1d4-4741582132bb /127.0.0.1 1 consumer1 my-topic(2) consumer7-c1d056fa-eabc-42ed-b557-c848f747ec49 /127.0.0.1 1 consumer7 16578 1 33155 my-second-topic(0) |
–describe --offsets
The new --offsets
option, that is allowed with --describe
only, produces the same result as the --describe
option itself. In other words, --offsets
is the default sub-option of --describe
:
Code Block | ||
---|---|---|
| ||
$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group --offsets Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers). TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID 16577 consumer1-81fe8a2a-0b29-4964-a1d4-4741582132bb /127.0.0.1 consumer1 my-topic HOST 2 16577 CLIENT-ID my-topic 33155 16578 consumer2-db6f12f2-e1de-4bb7-93ec-6170fefa8830 /127.0.0.10 16577 33154 consumer2 my-second-topic 16577 consumer1-81fe8a2a-0b29-4964-a1d4-4741582132bb 0 /127.0.0.1 33023 44543 consumer1 my-topic 11520 consumer7-c1d056fa-eabc-42ed-b557-c848f747ec49 /127.0.0.1 1 16578 consumer7 my-other-topic 33155 16577 1 consumer1-81fe8a2a-0b29-4964-a1d4-4741582132bb /127.0.0.1 7901 8652 consumer1 my-topic 751 - 2 16577 33155 16578 -consumer2-db6f12f2-e1de-4bb7-93ec-6170fefa8830 /127.0.0.1 -consumer2 my-othersecond-topic 0 0 33023 7902 44543 8654 11520 752 consumer7-c1d056fa-eabc-42ed-b557-c848f747ec49 /127.0.0.1 - consumer7 my-other-topic 1 7901 - 8652 751 - |
–describe --state
The --state
option, that can also be used with --describe
only, reports the group level information that comes back in the response of DescribeGroups
API call:
Code Block | ||
---|---|---|
| ||
$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group --state Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers). PROTOCOL:- range PROTOCOL-TYPE: consumer STATE: Stable |
--describe --subscription
The --subscription
option is also a sub-option of --describe
. It is used to extract subscription information of each member in the group. For each member, a comma-separated list of topics the member is subscribed to is returned:
Code Block | ||
---|---|---|
| ||
$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group --subscription Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers). CONSUMER-ID - - my-other-topic 0 7902 8654 752 - HOST CLIENT-ID SUBSCRIPTION consumer1-81fe8a2a-0b29-4964-a1d4-4741582132bb /127.0.0.1 - |
–describe --state
The --state
option, that can also be used with --describe
only, reports the group level information that comes back in the response of DescribeGroups
API call along with the coordinator id of the group:
Code Block | ||
---|---|---|
| ||
$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group --state Note: This consumer1will only show information about consumers that use the Java consumer API my-topic consumer2-db6f12f2-e1de-4bb7-93ec-6170fefa8830 /127.0.0.1 (non-ZooKeeper-based consumers). COORDINATOR (ID) ASSIGNMENT-STRATEGY STATE consumer2 #MEMBERS localhost:9092 (0) range my-topic consumer7-c1d056fa-eabc-42ed-b557-c848f747ec49 /127.0.0.1 Stable consumer7 my-second-topic |
...
3 |
In the above outputs, the width of the following columns will be dynamically determined (as opposed to the fixed widths that have been applied so far):
TOPIC
CONSUMER-ID
HOST
CLIENT-ID
COORDINATOR (ID)
Compatibility, Deprecation, and Migration Plan
...
Users who use the new-consumer based consumer groups and somehow rely on the output of the --describe
option of the ConsumerGroupCommand
may have to adjust their clients to use one of the new options provided: --describe
, –describe --members
, or –describe --partitions
; for example, in case they rely on the fact the command currently returns members with no assignment. Hopefully, how they use the command output will be simplified with these new options.
...