Status

Current state:  DraftDiscussion

Discussion thread:  here https://lists.apache.org/thread/7xn8bd6obobnr1j284d13r993cb4nlt8

Vote thread:  here

JIRA: https://issues.apache.org/jira/browse/KAFKA-20402

...

Kafka clients already provide rebalance-related callbacks, but relying on client-side behavior is operationally is operationally fragile. In practice, callback availability and behavior depend on SDK adoption and upgrade cycles, which are difficult to control sometimes. Even when teams implemented the callback with log only. users may reduce or disable the logs by mistake or some other reasons. This creates a common failure mode: when incidents occur, critical rebalance evidence is missing at the client side, making diagnosis slow and uncertain so that we have to query the log in kafka broker side.

However, today server-side observability side observability is limited: coordinator logs can indicate rebalance activity, but logs alone are not a reliable integration surface for automated handling, and existing  existing metrics do not carry the not carry the most actionable identifier (consume group id). As a result, operators can observe aggregate rebalance trends, but cannot lightweightly attach custom processing to specific groups without log parsing or invasive client changes.

This KIP proposes a lightweight broker-side rebalance callback capability to expose key rebalance context (including consume group id) on Kafka broker. 

In short, the goal is to provide provide a stable, centrally controlled extension controlled extension point for observability observability and operational automation, without forcing client SDK client SDK upgrades and without introducing without introducing high-cardinality metric cardinality metric tags. 

With the interface. New usage can be exposed . For example: AI trouble shooting for consume

...

• A list of fully qualified class names implementing ConsumerGroupRebalanceListener.
• Classes are instantiated and configured by the broker at startup.
• Empty value means callback feature is disabled

Note:  Dynamic updatesDynamic updates are not planed in this scope out for now so that we can keep the KIP small, low-risk, and validate real usage first.

...

Refer to the PR example:

Note:  Cover all non-classic consume group due to the classic consume group will be deprecated soon.

Compatibility, Deprecation, and Migration Plan 

...

Enhancing existing coordinator metrics is not sufficient for this use case because the key troubleshooting dimension is group.id, while server metrics are intentionally aggregate. If we add group.id as a metric label/tag, it introduces high-cardinality time series (potentially thousands of groups), which increases memory and storage cost on both broker and monitoring systems memory cost on broker and can degrade observability kafka performance.