THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation
The consumer group command, when used for describing a consumer group in the default or --offsets mode, currently reports current offset, log end offset, and the lag for each topic partition the group has offset for. It would be useful to extend the tool to also report the beginning offset for each partition, and the partition size. They could help with determining where the group stands with respect to consuming from the partition, whether it is consuming OK or is falling behind, what percentage of messages are consumed, whether the partition is empty, etc. The beginning or earliest log offset is not something that can be derived from the provided info. The partition size can be derived from beginning and end offset, but still is a useful metric.
Public Interfaces
The consumer group tool will be updated for --describe and --describe --offsets options by adding a LOG-START-OFFSET and PARTITION-SIZE columns column.
Proposed Changes
No protocol change is required as the log start offset is readily available (similar to how log end offset is retrieved); and the partition size is simply the difference between beginning and end offsets. Note that for a compacted topic the partition size cannot reliably be calculated from beginning and end offsets. For these partitions a compacted value appears under the PARTITION-SIZE column. Whether a topic is compacted or not can be determined using AdminClient.describeConfigs(...).As a result, the . The output of the command for --describe and --describe --offsets options will change from something like this:
| Code Block | ||
|---|---|---|
| ||
$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID my-topic 0 16577 33154 16577 consumer1-81fe8a2a-0b29-4964-a1d4-4741582132bb /127.0.0.1 consumer1 my-topic 1 16578 33155 16577 consumer1-81fe8a2a-0b29-4964-a1d4-4741582132bb /127.0.0.1 consumer1 my-topic 2 16577 33155 16578 consumer2-db6f12f2-e1de-4bb7-93ec-6170fefa8830 /127.0.0.1 consumer2 my-compactedsecond-topic 0 33023 44543 11520 consumer7-c1d056fa-eabc-42ed-b557-c848f747ec49 /127.0.0.1 consumer7 my-other-topic 1 7901 8652 751 - - - my-other-topic 0 7902 8654 752 - - - |
...
| Code Block | ||
|---|---|---|
| ||
$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group TOPIC PARTITION CURRENT-OFFSET LOG-START-OFFSET LOG-END-OFFSET PARTITION-SIZE LAG CONSUMER-ID HOST CLIENT-ID my-topic 0 16577 9856 33154 23298 16577 consumer1-81fe8a2a-0b29-4964-a1d4-4741582132bb /127.0.0.1 consumer1 my-topic 1 16578 9828 33155 23327 16577 consumer1-81fe8a2a-0b29-4964-a1d4-4741582132bb /127.0.0.1 consumer1 my-topic 2 16577 9847 33155 23308 16578 consumer2-db6f12f2-e1de-4bb7-93ec-6170fefa8830 /127.0.0.1 consumer2 my-compactedsecond-topic 0 33023 18484 44543 compacted 11520 consumer7-c1d056fa-eabc-42ed-b557-c848f747ec49 /127.0.0.1 consumer7 my-other-topic 1 7901 0 8652 8652 751 - - - my-other-topic 0 7902 0 8654 8654 752 - - - |
Compatibility, Deprecation, and Migration Plan
Users who use the consumer group command (e.g. , in custom tools) and somehow rely on parsing the output of the --describe or --describe --offsets options of the command may have to adjust their clients to account for these two this new columnscolumn.
Rejected Alternatives
N/A.