Status

Current state: "Under Discussion"

Discussion thread: 

JIRA:   KAFKA-19566 - Getting issue details... STATUS

Motivation

In KAFKA-18225, we identified that ClientQuotaCallback#updateClusterMetadata is not supported in KRaft mode, even though it is supported in Zookeeper mode. In Kafka 4.0, we addressed this gap by implementing the missing functionality; however, some limitations still remain:

1. In ClientQuotaCallback#updateClusterMetadata, we pass a Cluster object to this method. The Cluster is an immutable object that holds a lot of information, and a new Cluster is created every time there is a change in metadata. For large clusters with many partitions, this can result in significant memory pressure on Kafka. (see KAFKA-18239)
2. Some information in the Cluster object is unnecessary or confusing. For instance, Cluster#controller refers to a random broker node in a KRaft cluster, which is retained for backward compatibility as it represents the Zookeeper controller. Furthermore, in Zookeeper mode, we parse the listener using the listener name from the request. However, in KRaft mode, there is no listener name in the updateClusterMetadata path. As a result, the callback may receive multiple partition info entries for each listener name. (See KAFKA-19122).

To address the above issues, we proposed an enhancement in KIP-1162: Redesign ClientQuotaCallback#updateClusterMetadata. While it introduces several API changes, considering that this method has remained unimplemented for years since the introduction of KRaft—and no users have raised concerns—it doesn’t seem worthwhile to invest significant effort into it. However, leaving a non-functional API in place is still awkward, and we should discourage its continued use. Therefore, we propose deprecating the updateClusterMetadata method, and remove it entirely in Kafka 5.0.

Public Interfaces

Deprecating ClientQuotaCallback#updateClusterMetadata

public interface ClientQuotaCallback extends Configurable {
	...
     /**
     * This callback is invoked whenever there are changes in the cluster metadata, such as 
     * brokers being added or removed, topics being created or deleted, or partition leadership updates.
     * This is useful if quota computation takes partitions into account.
     * Topics that are being deleted will not be included in `cluster`.
     *
	 * @deprecated since 4.2 and should not be used any longer.
     * @param cluster Cluster metadata including partitions and their leaders if known
     * @return true if quotas have changed and metric configs may need to be updated
     */
	@Deprecated(since = "4.2", forRemoval = true)
    boolean updateClusterMetadata(Cluster cluster);
	...
}

Proposed Changes

Deprecating ClientQuotaCallback#updateClusterMetadata. No alternative method will be introduced, as there is currently no known demand for this functionality.

Compatibility, Deprecation, and Migration Plan

This KIP does not introduce any alternative methods, nor does it remove existing functionality—everything remains fully backward compatible. The ClientQuotaCallback#updateClusterMetadata method will be deprecated now and scheduled for removal in Kafka 5.0.

Test Plan

We will rely on the existing CustomQuotaCallbackTest.java to ensure the deprecated method still works.

Rejected Alternatives

• While KIP-1162 proposes a redesign, we’re letting updateClusterMetadata take a nap for now—but if a real use case shows up, we’re ready to wake it up and bring it back!