...
- The output of
--describe will change to return one row per topic partition in the group. This means that group members with no assigned partition will not be present in the default output. - A new switch
--members will be introduced, that can only be provided along with --describe, to return a list of all active members of the group. For each member the number of assigned topic partitions (without any offset information) and the assignment strategy used by the member are also returned. - A new switch
--offsets will be introduced, that can only be provided along with --describe, that returns the same default output of --describe. I.e., this will be the default switch that can be additionally used with --describe. - A new switch
--state will be introduced, that can only be provided along with --describe, to return group-level status information that is returned from the DescribeGroups API call. - A new switch
--subscription will be introduced, that can only be provided along with --describe, to list the topics each member is subscribed to.
Proposed Changes
To describe the proposed changes let us revisit the example above. This KIP proposes to change the --describe option of the ConsumerGroupCommand according to the following:
...
This option of the command returns offset and consumer information for each topic partition in the group. This is how the output of the example above would look like:
| Code Block |
|---|
|
$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group
Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
my-topic 0 16577 33154 16577 consumer1-81fe8a2a-0b29-4964-a1d4-4741582132bb /127.0.0.1 consumer1
my-topic 1 16578 33155 16577 consumer1-81fe8a2a-0b29-4964-a1d4-4741582132bb /127.0.0.1 consumer1
my-topic 2 16577 33155 16578 consumer2-db6f12f2-e1de-4bb7-93ec-6170fefa8830 /127.0.0.1 consumer2
my-second-topic 0 33023 44543 11520 consumer7-c1d056fa-eabc-42ed-b557-c848f747ec49 /127.0.0.1 consumer7
my-other-topic 1 7901 8652 751 - - -
my-other-topic 0 7902 8654 752 - - - |
...
A new option --members is introduced that is allowed with --describe only and returns info about active members of the group along with their assigned topic partitions:
| Code Block |
|---|
|
$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group --members
Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).
CONSUMER-ID HOST CLIENT-ID #PARTITIONS ASSIGNMENT-STRATEGY
consumer1-81fe8a2a-0b29-4964-a1d4-4741582132bb /127.0.0.1 consumer1 2 range
consumer2-db6f12f2-e1de-4bb7-93ec-6170fefa8830 /127.0.0.1 consumer2 1 range, round-robin
consumer7-c1d056fa-eabc-42ed-b557-c848f747ec49 /127.0.0.1 consumer7 1 range |
–describe --offsets
The new --offsets option, that is allowed with --describe only, produces the same result as the --describe option itself. In other words, --offsets is the default sub-option of --describeThere is an additional --verbose switch that can be used along with this option to include additional details about each group member:
| Code Block |
|---|
|
$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group --offsetsmembers --verbose
Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).
TOPICCONSUMER-ID PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG HOST CONSUMER-ID CLIENT-ID HOST #PARTITIONS ASSIGNMENT-STRATEGY SUBSCRIPTION CLIENT-ID
my-topicASSIGNMENT
consumer1-81fe8a2a-0b29-4964-a1d4-4741582132bb /127.0.0.1 0 consumer1 16577 2 33154 range 16577 consumer1-81fe8a2a-0b29-4964-a1d4-4741582132bb /127.0.0.1 my-topic consumer1
my-topic (0,1)
consumer2-db6f12f2-e1de-4bb7-93ec-6170fefa8830 /127.0.0.1 consumer2 1 1 16578 range, round-robin 33155 my-topic 16577 consumer1-81fe8a2a-0b29-4964-a1d4-4741582132bbmy-topic(2)
consumer7-c1d056fa-eabc-42ed-b557-c848f747ec49 /127.0.0.1 consumer1
my-topicconsumer7 1 2 range 16577 33155 16578 consumer2-db6f12f2-e1de-4bb7-93ec-6170fefa8830 /127.0.0.1my-second-topic my-second-topic(0) |
–describe --offsets
The new --offsets option, that is allowed with --describe only, produces the same result as the --describe option itself. In other words, --offsets is the default sub-option of --describe:
| Code Block |
|---|
|
$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group --offsets
Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
my-topic 0 16577 33154 16577 consumer1-81fe8a2a-0b29-4964-a1d4-4741582132bb /127.0.0.1 consumer2
my-second-topic 0 consumer1
my-topic 33023 44543 1 11520 16578 consumer7-c1d056fa-eabc-42ed-b557-c848f747ec49 /127.0.0.1 33155 16577 consumer7
my-other-topicconsumer1-81fe8a2a-0b29-4964-a1d4-4741582132bb /127.0.0.1 1 consumer1
my-topic 7901 8652 2 751 16577 - 33155 16578 consumer2-db6f12f2-e1de-4bb7-93ec-6170fefa8830 /127.0.0.1 consumer2
my-second-topic - 0 33023 -
my-other-topic 44543 11520 0 consumer7-c1d056fa-eabc-42ed-b557-c848f747ec49 7902/127.0.0.1 8654 consumer7
my-other-topic 752 - 1 7901 8652 751 - - |
–describe --state
The --state option, that can also be used with --describe only, reports the group level information that comes back in the response of DescribeGroups API call:
| Code Block |
|---|
|
$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group --state
Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).
PROTOCOL:- range
PROTOCOL-TYPE: consumer
STATE: Stable |
--describe --subscription
The --subscription option is also a sub-option of --describe. It is used to extract subscription information of each member in the group. For each member, a comma-separated list of topics the member is subscribed to is returned:
| Code Block |
|---|
|
$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group --subscription
Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).
CONSUMER-ID -
my-other-topic 0 7902 8654 752 - HOST CLIENT-ID SUBSCRIPTION
consumer1-81fe8a2a-0b29-4964-a1d4-4741582132bb /127.0.0.1 consumer1 my-topic
consumer2-db6f12f2-e1de-4bb7-93ec-6170fefa8830 /127.0.0.1 consumer2 my-topic
consumer7-c1d056fa-eabc-42ed-b557-c848f747ec49 /127.0.0.1- |
–describe --state
The --state option, that can also be used with --describe only, reports the group level information that comes back in the response of DescribeGroups API call:
| Code Block |
|---|
|
$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group --state
Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).
ASSIGNMENT-STRATEGY STATE #MEMBERS
range consumer7 Stable my-second-topic |
...
Compatibility, Deprecation, and Migration Plan
...