This page is meant as a template for writing a KIP. To create a KIP choose Tools->Copy on this page and modify with your content and replace the heading with the next KIP number and a description of your issue. Replace anything in italics with your own description.

Status

Current state: [One of "Under Discussion", "Accepted", "Rejected"]"Draft"

Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]

JIRA: here [Change the link from KAFKA-56391 to your own ticket]

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Describe the problems you are trying to solve.

Public Interfaces

Briefly list any new interfaces that will be introduced as part of this proposal or any existing interfaces that will be removed or changed. The purpose of this section is to concisely call out the public contract that will come along with this feature.

A public interface is any change to the following:

• Binary log format

• The network protocol and api behavior

• Any class in the public packages under clientsConfiguration, especially client configuration

  • org/apache/kafka/common/serialization

  • org/apache/kafka/common

  • org/apache/kafka/common/errors

  • org/apache/kafka/clients/producer

  • org/apache/kafka/clients/consumer (eventually, once stable)

• Monitoring

• Command line tools and arguments

• Anything else that will likely break existing users in some way when they upgrade

Proposed Changes

...

The [DescribeGroups|https://kafka.apache.org/protocol#The_Messages_DescribeGroups] protocol v1 currently returns this information for each consumer group:

• error_code
• group_id
• state
• protocol_type
• protocol
• members (only a member summary is returned per member that misses some useful member info too)

There are additional info in a GroupMetadata object on the server side, some of which could be useful if exposed via the DescribeGroups protocol. Here are some examples:

• generationId
• leaderId
• newMemberAdded

• receivedConsumerOffsetCommits

• receivedTransactionalOffsetCommits

• supportedProtocols per member

Enhancing the protocol with this additional info means improving the existing tools that make use of it. For example, using this additional info, the consumer group command's {{\-\-describe}} output will provide more information about each consumer group to help with its monitoring / troubleshooting / ....

Public Interfaces

This is the latest version (v1) of DescribeGroups protocol:

DescribeGroups Request (Version: 1) => [group_ids] 
  group_ids => STRING
 
DescribeGroups Response (Version: 1) => throttle_time_ms [groups] 
  throttle_time_ms => INT32
  groups => error_code group_id state protocol_type protocol [members] 
    error_code => INT16
    group_id => STRING
    state => STRING
    protocol_type => STRING
    protocol => STRING
    members => member_id client_id client_host member_metadata member_assignment 
      member_id => STRING
      client_id => STRING
      client_host => STRING
      member_metadata => BYTES
      member_assignment => BYTES

The suggestion is to enhance the protocol to version 2 and add some of these currently missing information (* indicates suggested fields to be added to the protocol):

DescribeGroups Request (Version: 2) => [group_ids] 
  group_ids => STRING


DescribeGroups Response (Version: 2) => throttle_time_ms [groups] 
  throttle_time_ms => INT32
  groups => error_code group_id state protocol_type protocol generation_id leader_id new_member_added received_consumer_offset_commits received_transactional_offset_commits [members] 
    error_code => INT16
    group_id => STRING
    state => STRING
    protocol_type => STRING
    protocol => STRING
  * generation_id => INT32
  * leader_id => STRING
  * new_member_added => BOOLEAN
  * received_consumer_offset_commits => BOOLEAN
  * received_transactional_offset_commits => BOOLEAN
    members => member_id client_id client_host member_metadata member_assignment [member_protocols]
      member_id => STRING
      client_id => STRING
      client_host => STRING
      member_metadata => BYTES
      member_assignment => BYTES
  *   member_protocols => STRING

Proposed Changes

The proposal is to

1. Create version 2 of DescribeGroups protocol to include additional information about the consumer group and also each group member for the API clients.
2. Update the consumer group command to provide the added information in the command output (where applicable). Note that this KIP will impact KIP-175 where group state and detailed member information will also be returned by the consumer group command using additional command line options. Therefore,
   1. If this KIP is implemented before KIP-175, this step can be skipped (as KIP-175 will make the command line adjustments to leverage newly available information).
   2. If this KIP is implemented after KIP-175, this step will be implemented as part of the KIP.

Compatibility, Deprecation, and Migration Plan

• What impact (if any) will there be on existing users?
• If we are changing behavior how will we phase out the older behavior?
• If we need special migration tools, describe them here.
• When will we remove the existing behavior?

Rejected Alternatives

If this KIP is implemented before KIP-175, there is no compatibility concerns. This all existing protocol fields remain unchanged.

If the KIP is implemented after KIP-175, the output for sub-options --members and --state that are introduced in that KIP will be modified to also include the newly added fields. Therefore, clients who rely on the output for those sub-options may need to be adjusted.

Rejected Alternatives

 If there are alternative ways of accomplishing the same thing, what were they? The purpose of this section is to motivate why the design is the way it is and not some other way.