You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 6 Next »


Status

Current state: Under Discussion

Discussion thread: here

JIRA: Unable to render Jira issues macro, execution error.

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

The consumer group command, when used for describing a consumer group in the default or --offsets mode, currently reports current offset, log end offset, and the lag for each topic partition the group has offset for. It would be useful to extend the tool to also report the beginning offset for each partition, and the partition size. The beginning or earliest log offset is not something that can be derived from the provided info. The partition size can be derived from beginning and end offset, but still is a useful metric.

Public Interfaces

The consumer group tool will be updated for --describe and --describe --offsets options by adding LOG-START-OFFSET and PARTITION-SIZE columns.


Proposed Changes

No protocol change is required as the log start offset is readily available (similar to how log end offset is retrieved); and the partition size is simply the difference between beginning and end offsets. Note that for a compacted topic the partition size cannot reliably be calculated from beginning and end offsets. For these partitions a compacted value appears under the PARTITION-SIZE column. Whether a topic is compacted or not can be determined using AdminClient.describeConfigs(...).

As a result, the output of the command for --describe and --describe --offsets options will change from something like this:

$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group
TOPIC                          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG        CONSUMER-ID                                       HOST                           CLIENT-ID
my-topic                       0          16577           33154           16577      consumer1-81fe8a2a-0b29-4964-a1d4-4741582132bb    /127.0.0.1                     consumer1
my-topic                       1          16578           33155           16577      consumer1-81fe8a2a-0b29-4964-a1d4-4741582132bb    /127.0.0.1                     consumer1
my-topic                       2          16577           33155           16578      consumer2-db6f12f2-e1de-4bb7-93ec-6170fefa8830    /127.0.0.1                     consumer2
my-compacted-topic             0          33023           44543           11520      consumer7-c1d056fa-eabc-42ed-b557-c848f747ec49    /127.0.0.1                     consumer7
my-other-topic                 1          7901            8652            751        -                                                 -                              -
my-other-topic                 0          7902            8654            752        -                                                 -                              -

to something like this:

$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group
TOPIC                          PARTITION  CURRENT-OFFSET  LOG-START-OFFSET  LOG-END-OFFSET  PARTITION-SIZE  LAG        CONSUMER-ID                                       HOST                           CLIENT-ID
my-topic                       0          16577           9856              33154           23298           16577      consumer1-81fe8a2a-0b29-4964-a1d4-4741582132bb    /127.0.0.1                     consumer1
my-topic                       1          16578           9828              33155           23327           16577      consumer1-81fe8a2a-0b29-4964-a1d4-4741582132bb    /127.0.0.1                     consumer1
my-topic                       2          16577           9847              33155           23308           16578      consumer2-db6f12f2-e1de-4bb7-93ec-6170fefa8830    /127.0.0.1                     consumer2
my-compacted-topic             0          33023           18484             44543           compacted       11520      consumer7-c1d056fa-eabc-42ed-b557-c848f747ec49    /127.0.0.1                     consumer7
my-other-topic                 1          7901            0                 8652            8652            751        -                                                 -                              -
my-other-topic                 0          7902            0                 8654            8654            752        -                                                 -                              -


Compatibility, Deprecation, and Migration Plan

Users who use the consumer group command (e.g., in custom tools) and somehow rely on parsing the output of the --describe or --describe --offsets options of the command may have to adjust their clients to account for these two new columns.

Rejected Alternatives

N/A.

  • No labels