You are viewing an old version of this page. View the current version.

Compare with Current View Page History

Version 1 Next »

Status

Current state: Under Discussion

Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]

JIRA: Unable to render Jira issues macro, execution error.

Motivation

1) Running a set of commands like `--describe`, `--delete` or `--reset-offsets` for every single consumer group appears to be quite a tedios, time and memory consuming task. Each time you have to start a separate JVM to perform a query for a single group or spend some time writing your own client code. 

2) Quite often one would like to perform operations on all consumer groups, thus, it would be great to have something as `--all-groups` option, similar to `--all-topics`.  

3) In other cases, you would like to perform operations on a certain subset of consumer groups, thus, it would be great to have something as a repeatable `--group` option for each group.   

4) By performing a Describe query on all consumer groups with a combination of options `--describe` and `--state` will allow us to grep consumer groups by their state (like `Stable, Dead, Empty, PreparingRebalance, NonExisting`). The only thing we have to do is to extend the "results table" with a GROUP column. 

5) Support new CSV file format to export consumer offsets.

Public Interfaces

1) Providing the command line tool kafka-consumer-groups.sh with support of repeating --group parameter for operations --describe, --delete, --reset-offsets:

Usage example:

bin/kafka-consumer-groups.sh --bootstrap-server :9092 --describe      --group group-1 --group group-2 --group group-3
bin/kafka-consumer-groups.sh --bootstrap-server :9092 --delete        --group group-1 --group group-2 --group group-3
bin/kafka-consumer-groups.sh --bootstrap-server :9092 --reset-offsets --group group-1 --group group-2 --group group-3 --topic test

2) Providing the command line tool kafka-consumer-groups.sh with an optional --all-groups parameter for operations --describe, --delete, --reset-offsets: 

parameter name

defaults to

type

usage

--all-groupsinactiveflagoptional

Usage example:

bin/kafka-consumer-groups.sh --bootstrap-server :9092 --describe      --all-groups
bin/kafka-consumer-groups.sh --bootstrap-server :9092 --delete        --all-groups
bin/kafka-consumer-groups.sh --bootstrap-server :9092 --reset-offsets --all-groups --topic test


3) Extending data output table format by adding "GROUP" column for Describe queries:

--describe:

TOPIC

PARTITION

CURRENT-OFFSET

LOG-END-OFFSET

LAGCONSUMER-IDHOSTCLIENT-IDGROUP









--describe --members:

CONSUMER-IDHOSTCLIENT-ID#PARTITIONSGROUP





--describe --members --verbose

CONSUMER-IDHOSTCLIENT-ID#PARTITIONSGROUPASSIGNMENT






--describe --state

COORDINATOR (ID)ASSIGNMENT-STRATEGYSTATE#MEMBERSGROUP






4) Update CSV file to support multiple consumer groups. Thus, we will have to update table format by adding a consumer group column. 

New CSV file format:

GROUPTOPICPARTITIONOFFSET





Proposed Changes

  • Providing the command line tool kafka-consumer-groups.sh with support of repeating --group parameter for operations --describe, --delete, --reset-offsets
  • Providing the command line tool kafka-consumer-groups.sh with an optional --all-groups parameter for operations --describe, --delete, --reset-offsets
  • Extending data output table format by adding "GROUP" column for all --describe queries
  • Update CSV file to support multiple consumer groups.
  • We will also have to make sure CSV generation is done properly since there are no restrictions on consumer group names and symbols like commas and quotes are allowed. That is, CSV file generation with export/import of consumer offsets needs some rework. 

Compatibility, Deprecation, and Migration Plan

  • Positive impact on existing users
  • We are not changing behavior but adding new features

Rejected Alternatives

None.

  • No labels