Table of Contents |
---|
Status
Current state: Under Discussion Adopted
Discussion thread: here
JIRA: Jira server ASF JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key KAFKA-5526
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
...
As far as describing a consumer group goes, the ConsumerGroupCommand
currently lists all members of the group along with associated topic partitions and their committed offsets all all in one view. For example:
Code Block | ||
---|---|---|
| ||
$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers). TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID my-topic 0 1 16577 30857 33154 118320 8746316577 consumer2consumer1-5d3a62da81fe8a2a-abe10b29-427d4964-a9e9a1d4-4962c35f2dae4741582132bb /127.0.0.1 consumer2consumer1 my-topic 1 2 16578 30857 33155 118320 8746316577 consumer3consumer1-ad71338d81fe8a2a-d3790b29-4cff4964-be2da1d4-1e3971f32f514741582132bb /127.0.0.1 consumer3consumer1 my-topic 2 0 16577 31357 33155 118320 8696316578 consumer1consumer2-5e864292db6f12f2-bef0e1de-45f14bb7-9b9693ec-ea56943609e76170fefa8830 /127.0.0.1 consumer1consumer2 my-second-topic 0 33023 - 44543 - 11520 - - consumer4-e173f09d-c761-4f4e-95c7-6fb73bb8fbffconsumer7-c1d056fa-eabc-42ed-b557-c848f747ec49 /127.0.0.1 consumer4consumer7 - - - - - consumer5consumer8-7b80e4286078de6a-f8ffaf63-43f34c7a-83608bad-afd1c8ba43eaf03cb3606075 /127.0.0.1 consumer5consumer8 my-other-topic 1 68957901 68958652 0751 - - - my-other-topic 0 68947902 68948654 0 752 - - - |
The information included in the command output includes:
...
This KIP suggests breaking down the command to cover each some of the information pieces above separately using additional switches. There are a couple of reasons behind this improvement:
- The above output, with all the missing column values, looks weird and could be difficult to interpret. For a big group, where there are many consumers without assigned partition, or there are many partitions without a consumer, the command will output a long list, while the user may only be interested in, for example, active members only.
- A user is typically interested in one of the information pieces above when issuing the command: either offset information, or members, or topic partitions associated with the group.
As part of this KIP, additional information about the group will also be generated that is currently missing. For example, generation id of the group, or the assignment policy used by each member.
Public Interfaces
The change required for this includes updating the ConsumerGroupCommand
tool:
- The output of
--describe
will change to return only rows for active consumers one row per topic partition in the group. This means that group members with assigned topic partitionsno assigned partition will not be present in the default output. - A new switch
--members
will be introduced, that can only be provided along with--describe
, to return a list of all active members of the group. For each member the number of assigned topic partitions (without any offset information) and the assignment strategy used by the member are also returned. Additional information (such as member subscription and assignment) can also be provided with a--members
--verbose
option. - A new switch
--partitions
willoffsets
will be introduced, that can only be provided along with--describe
, to return a list of all topic partitions with committed offsets in the group. For each topic partition, current offset, log end offset, and lag are also returned.that returns the same default output of--describe
. I.e., this will be the default switch that can be additionally used with--describe
. - A new switch
--state
will be introduced, that can only be provided along with--describe
, to return group-level status information that is returned from theDescribeGroups
API call.
Proposed ChangesProposed Changes
To describe the proposed changes let us revisit the example above. This KIP proposes to change the --describe
option of the ConsumerGroupCommand
according to the following:
...
This option of the command returns information on only active members of the group with assigned partitionsoffset and consumer information for each topic partition in the group. This is how the output of the example above would look like:
Code Block | ||
---|---|---|
| ||
$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers). TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID my-topic 10 3085716577 11832033154 87463 16577 consumer2 consumer1-5d3a62da81fe8a2a-abe10b29-427d4964-a9e9a1d4-4962c35f2dae4741582132bb /127.0.0.1 consumer2consumer1 my-topic 1 2 16578 30857 33155 118320 8746316577 consumer3consumer1-ad71338d81fe8a2a-d3790b29-4cff4964-be2da1d4-1e3971f32f514741582132bb /127.0.0.1 consumer3consumer1 my-topic 2 0 16577 31357 33155 118320 8696316578 consumer1consumer2-5e864292db6f12f2-bef0e1de-45f14bb7-9b9693ec-ea56943609e76170fefa8830 /127.0.0.1 consumer1 |
--describe --members
A new option --members
is introduced that is allowed with --describe
only and returns info about active members of the group along with their assigned topic partitions:
Code Block | ||
---|---|---|
| ||
$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group --members Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers). CONSUMER-ID consumer2 my-second-topic 0 33023 44543 11520 consumer7-c1d056fa-eabc-42ed-b557-c848f747ec49 /127.0.0.1 HOST consumer7 my-other-topic CLIENT-ID 1 7901 TOPIC(PARTITIONS) consumer2-5d3a62da-abe1-427d-a9e9-4962c35f2dae /127.0.0.1 8652 751 consumer2 - my-topic(1) consumer3-ad71338d-d379-4cff-be2d-1e3971f32f51 /127.0.0.1 - consumer3 my-topic(2) consumer1my-5e864292-bef0-45f1-9b96-ea56943609e7other-topic /127.0.0.1 0 7902 consumer1 8654 my-topic(0) |
Note that the TOPIC(PARTITIONS)
column contains a comma separate list of topics the member is consuming from and corresponding partitions of each topic appear in parentheses (also in comma separated format). For example: my-topic(0,2),my-other-topic(1,3)
...
752 - - - |
Note that consumers without any assigned partitions will no longer appear in the listing.
--describe --members
and --describe
--members
--verbose
A new option --partitionsmembers
is introduced that is allowed with --describe
only and for each topic partition the group has stored offset for it reports the current offset, log end offset, and the lag:returns info about active members of the group along with their assigned topic partitions:
Code Block | ||
---|---|---|
| ||
$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group --members
Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).
CONSUMER-ID HOST CLIENT-ID #PARTITIONS
consumer1-81fe8a2a-0b29-4964-a1d4-4741582132bb /127.0.0.1 consumer1 2
consumer2-db6f12f2-e1de-4bb7-93ec-6170fefa8830 /127.0.0.1 consumer2 1
consumer7-c1d056fa-eabc-42ed-b557-c848f747ec49 /127.0.0.1 consumer7 1 |
There is an additional --verbose
switch that can be used along with this option to include additional details about each group member:
Code Block | ||
---|---|---|
| ||
$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group --members --verbose
Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).
CONSUMER-ID HOST CLIENT-ID #PARTITIONS ASSIGNMENT
consumer1-81fe8a2a-0b29-4964-a1d4-4741582132bb /127.0.0.1 consumer1 2 my-topic(0,1)
consumer2-db6f12f2-e1de-4bb7-93ec-6170fefa8830 /127.0.0.1 consumer2 1 my-topic(2)
consumer7-c1d056fa-eabc-42ed-b557-c848f747ec49 /127.0.0.1 consumer7 1 my-second-topic(0) |
–describe --offsets
The new --offsets
option, that is allowed with --describe
only, produces the same result as the --describe
option itself. In other words, --offsets
is the default sub-option of --describe
:
Code Block | ||
---|---|---|
| ||
$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group --offsets
Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
my-topic 0 16577 33154 16577 consumer1-81fe8a2a-0b29-4964-a1d4-4741582132bb /127.0.0.1 consumer1
my-topic 1 16578 33155 16577 consumer1-81fe8a2a-0b29-4964-a1d4-4741582132bb /127.0.0.1 consumer1
my-topic 2 16577 33155 16578 consumer2-db6f12f2-e1de-4bb7-93ec-6170fefa8830 /127.0.0.1 consumer2
my-second-topic 0 33023 44543 11520 consumer7-c1d056fa-eabc-42ed-b557-c848f747ec49 /127.0.0.1 consumer7
my-other-topic 1 7901 8652 751 - | ||
Code Block | ||
| ||
$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group --partitions Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers). TOPIC - PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG my-topic - my-other-topic 1 30857 0 118320 7902 87463 my-topic 8654 752 2 - 30857 118320 87463 my-topic - 0 31357 118320 - |
–describe --state
The --state
option, that can also be used with --describe
only, reports the group level information that comes back in the response of DescribeGroups
API call along with the coordinator id of the group:
Code Block | ||
---|---|---|
| ||
$ 86963 my-other-topic 1 6895 bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group --state Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers). COORDINATOR (ID) ASSIGNMENT-STRATEGY 6895 STATE #MEMBERS localhost:9092 0 my-other-topic(0) range 0 Stable 6894 6894 03 |
In the above outputs, the width of the following columns will be dynamically determined (as opposed to the fixed widths that have been applied so far):
TOPIC
CONSUMER-ID
HOST
CLIENT-ID
COORDINATOR (ID)
Compatibility, Deprecation, and Migration Plan
...
Users who use the new-consumer based consumer groups and somehow rely on the output of the --describe
option of the ConsumerGroupCommand
may have to adjust their clients to use one of the new options provided: --describe
, –describe --members
, or –describe --partitions
; for example, in case they rely on the fact the command currently returns members with no assignment. Hopefully, how they use the command output will be simplified with these new options.
...