Current state: "Under Discussion"
Discussion thread: here
JIRA: KAFKA-19577
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Partition replicas placement is a important topic in Kafka. Depending on different use cases, users may prefer different placement strategies. Currently, Kafka uses the default StripedReplicaPlacer, which implement round-robin algorithm to distribute replicas. This approach works well for the majority of scenarios.
However, there are still special cases—such as internal testing or experimental features—where users may want to define a custom placement strategy. Therefore, we propose providing a mechanism that allows users to implement their own ReplicaPlacer.
Previously, KIP-660: Pluggable ReplicaPlacer proposed making the ReplicaPlacer pluggable, but the proposal was rejected. The main concern was that an intelligent placement strategy might require a full view of the cluster, which could be prohibitively expensive in large deployments. Furthermore, such implementations could become overly complex and might introduce latency that impacts the controller thread, thus affecting system stability.
Nevertheless, we can make the ReplicaPlacer an internal pluggable component. This means the mechanism would not be exposed to external users, but could be utilized in internal testing, special-purpose scenarios, or controlled experiments. By allowing different implementations to be injected internally, we can make the placement logic more flexible for testing purposes, without compromising the stability of the controller.
New broker configuration:
Add a new internal configuration: internal.replica.placer.class.name. This configuration allows users to define a custom ReplicaPlacer implementation.
public static final String INTERNAL_REPLICA_PLACER_CLASS_NAME = "internal.replica.placer.class.name";
public static final String INTERNAL_REPLICA_PLACER_CLASS_NAME_DEFAULT = StripedReplicaPlacer.class.getName();
public static final String INTERNAL_REPLICA_PLACER_CLASS_NAME_DOC = "The fully qualified class name of the ReplicaPlacer implementation. This class is used by the broker to determine the placement of replicas when topics or partitions are created."; |
This KIP adds internal config which doesn’t break compatibility.
Add new test for new internal config internal.replica.placer.class.name
n/a