Status

Current state: Voting In Progress

Discussion thread: here

Vote thread: here

JIRA: Unable to render Jira issues macro, execution error.

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

The consumer group command, when used for describing a consumer group in the default or --offsets mode, currently reports current offset, log end offset, and the lag for each topic partition the group has offset for. It would be useful to extend the tool to also report the beginning offset for each partition. They could help with determining where the group stands with respect to consuming from the partition, whether it is consuming OK or is falling behind, what percentage of messages are consumed, whether the partition is empty, etc. The beginning or earliest log offset is not something that can be derived from the provided info.

Public Interfaces

The consumer group tool will be updated for --describe and --describe --offsets options by adding a LOG-START-OFFSET column.


Proposed Changes

No protocol change is required as the log start offset is readily available (similar to how log end offset is retrieved). The output of the command for --describe and --describe --offsets options will change from something like this:

$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group
TOPIC                          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG        CONSUMER-ID                                       HOST                           CLIENT-ID
my-topic                       0          16577           33154           16577      consumer1-81fe8a2a-0b29-4964-a1d4-4741582132bb    /127.0.0.1                     consumer1
my-topic                       1          16578           33155           16577      consumer1-81fe8a2a-0b29-4964-a1d4-4741582132bb    /127.0.0.1                     consumer1
my-topic                       2          16577           33155           16578      consumer2-db6f12f2-e1de-4bb7-93ec-6170fefa8830    /127.0.0.1                     consumer2
my-second-topic                0          33023           44543           11520      consumer7-c1d056fa-eabc-42ed-b557-c848f747ec49    /127.0.0.1                     consumer7
my-other-topic                 1          7901            8652            751        -                                                 -                              -
my-other-topic                 0          7902            8654            752        -                                                 -                              -

to something like this:

$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group
TOPIC                          PARTITION  CURRENT-OFFSET  LOG-START-OFFSET  LOG-END-OFFSET  LAG        CONSUMER-ID                                       HOST                           CLIENT-ID
my-topic                       0          16577           9856              33154           16577      consumer1-81fe8a2a-0b29-4964-a1d4-4741582132bb    /127.0.0.1                     consumer1
my-topic                       1          16578           9828              33155           16577      consumer1-81fe8a2a-0b29-4964-a1d4-4741582132bb    /127.0.0.1                     consumer1
my-topic                       2          16577           9847              33155           16578      consumer2-db6f12f2-e1de-4bb7-93ec-6170fefa8830    /127.0.0.1                     consumer2
my-second-topic                0          33023           18484             44543           11520      consumer7-c1d056fa-eabc-42ed-b557-c848f747ec49    /127.0.0.1                     consumer7
my-other-topic                 1          7901            0                 8652            751        -                                                 -                              -
my-other-topic                 0          7902            0                 8654            752        -                                                 -                              -


Compatibility, Deprecation, and Migration Plan

Users who use the consumer group command (e.g. in custom tools) and somehow rely on parsing the output of the --describe or --describe --offsets options of the command may have to adjust their clients to account for this new column. 

Rejected Alternatives

N/A.

  • No labels