...

Apache Kafka supports rack-aware partition assignment to improve fault tolerance and reduce cross-rack network traffic/cost. However, the current Admin API does not expose rack information for all type of consumer group members, despite this information being available at the protocol level.

...

However, when users call AdminClient.describeConsumerGroups(), the returned MemberDescription objects do not include this rack information. The rack ID is available in the wire protocol but is discarded during response processing in DescribeConsumerGroupsHandler.

The Similar case is for AdminClient.describeShareGroups():


Problem Statement

This limitation creates several issues:

1. Monitoring and Observability: Operators cannot determine the rack distribution of consumer group members through the Admin API, making it difficult to verify that rack-aware assignment is working correctly.

2. Inconsistency: Other group types expose rack information:

   • StreamsGroupMemberDescription includes rackId() method
   • The underlying protocol supports it for consumer groups
   • Only the public Admin API omits this information
3. Diagnostics: When troubleshooting rack-aware assignment issues or network problems, operators need to use lower-level tools or custom code to access rack information that should be readily available.
4. Third-party Tools: Monitoring and management tools built on the Admin API cannot display rack information, limiting their usefulness for rack-aware deployments.

Take one example:  currently we have to implemen our AZ/Rack analysis using a workaround — passing the rack information into the clientId field and parsing it afterward.

kafkaConsumerConfig.customConfig(ConsumerConfig.CLIENT_ID_CONFIG, generateClientIdWithRack(ip, rack));

Image Added

Public Interfaces

add dedicated public interface for this case:

...