Table of Contents |
---|
This page is meant as a template for writing a KIP. To create a KIP choose Tools->Copy on this page and modify with your content and replace the heading with the next KIP number and a description of your issue. Replace anything in italics with your own description.
Status
Current state: [One of "Under Discussion", "Accepted", "Rejected"]"Draft"
Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]
JIRA: here [Change the link from KAFKA-56391 to your own ticket]
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation
Describe the problems you are trying to solve.
Public Interfaces
Briefly list any new interfaces that will be introduced as part of this proposal or any existing interfaces that will be removed or changed. The purpose of this section is to concisely call out the public contract that will come along with this feature.
A public interface is any change to the following:
Binary log format
The network protocol and api behavior
Any class in the public packages under clientsConfiguration, especially client configuration
org/apache/kafka/common/serialization
org/apache/kafka/common
org/apache/kafka/common/errors
org/apache/kafka/clients/producer
org/apache/kafka/clients/consumer (eventually, once stable)
Monitoring
Command line tools and arguments
- Anything else that will likely break existing users in some way when they upgrade
Proposed Changes
...
The DescribeGroups
protocol v1 currently returns this information for each consumer group:
error_code
group_id
state
protocol_type
protocol
members
(only a member summary is returned per member that misses some useful member info too)
Additional info exist or can be derived in a GroupMetadata
object on the server side, some of which could be useful if exposed via the DescribeGroups
protocol. Here are some examples:
generationId
leaderId
supportedProtocols
per member- subscribed topics per member
Enhancing the protocol with some of this additional info means improving the existing tools that make use of it. For example, using this additional info, the consumer group command's --describe
output will provide more information about each consumer group to help with understanding the group behavior, its monitoring, troubleshooting, etc.
Public Interfaces
This is the latest version (v1) of DescribeGroups
protocol:
Code Block |
---|
DescribeGroups Request (Version: 1) => [group_ids]
group_ids => STRING
DescribeGroups Response (Version: 1) => throttle_time_ms [groups]
throttle_time_ms => INT32
groups => error_code group_id state protocol_type protocol [members]
error_code => INT16
group_id => STRING
state => STRING
protocol_type => STRING
protocol => STRING
members => member_id client_id client_host member_metadata member_assignment
member_id => STRING
client_id => STRING
client_host => STRING
member_metadata => BYTES
member_assignment => BYTES |
The suggestion is to enhance the protocol to version 2 and add some of these currently missing information (*
indicates suggested fields to be added to the protocol):
Code Block |
---|
DescribeGroups Request (Version: 2) => [group_ids]
group_ids => STRING
DescribeGroups Response (Version: 2) => throttle_time_ms [groups]
throttle_time_ms => INT32
groups => error_code group_id state protocol_type protocol generation_id leader_id new_member_added received_consumer_offset_commits received_transactional_offset_commits [members]
error_code => INT16
group_id => STRING
state => STRING
protocol_type => STRING
protocol => STRING
* generation_id => INT32
* leader_id => STRING
members => member_id client_id client_host member_metadata member_assignment member_subscription [member_protocols]
member_id => STRING
client_id => STRING
client_host => STRING
member_metadata => BYTES
member_assignment => BYTES
* member_subscription => BYTE
* member_protocols => STRING |
Proposed Changes
The proposal is to
- Create version 2 of
DescribeGroups
protocol to include additional information about the consumer group and also each group member for the API clients. - Update the consumer group command to provide the added information in the command output (where applicable). Note that this KIP will improve upon KIP-175 where group state and detailed member information will also be returned by the consumer group command using additional command line options. Therefore, the output of some of the sub-options introduced in KIP-175 will improve to include the newly available information by this KIP. Suggested changes are:
- Updating the
--state
option to also include aGENERATION-ID
andLEADER-ID
column. - Providing the missing member
ASSIGNMENT-STRATEGY
value for--members
option. - Providing the missing member
SUBSCRIPTION
value for--members --verbose
option.
- Updating the
- On a related subject but perhaps unrelated to the protocol enhancement, also update the consumer group command to provide filtering by topic as well (similar to how the old consumer offset checker tool supported this filtering).
Compatibility, Deprecation, and Migration Plan
- What impact (if any) will there be on existing users?
- If we are changing behavior how will we phase out the older behavior?
- If we need special migration tools, describe them here.
- When will we remove the existing behavior?
Rejected Alternatives
...
Since this KIP follows KIP-175, the output for sub-options --members
and --state
that are introduced in that KIP will be modified to also include the newly added fields. Therefore, clients who rely on the output for those sub-options may need to be adjusted. It also helps the consumer group --describe
command to fill some of the newly added columns introduced by KIP-175 (i.e. ASSIGNMENT-STRATEGY
for --members
options, and SUBSCRIPTION
for --members --verbose
option).
Rejected Alternatives
- Providing this additional information in the server logs: It is much more convenient to look up this new pieces of information via the consumer group command tool. Plus, these fields and their underlying concepts are not likely to disappear in the future. So we would not have to worry about making a protocol change that will likely become obsolete in the future.