Status

Current state: "Under Discussion"

Discussion thread: here

JIRA: KAFKA-19577

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Partition replicas placement is a important topic in Kafka. Depending on different use cases, users may prefer different placement strategies. Currently, Kafka uses the default StripedReplicaPlacer, which implement round-robin algorithm to distribute replicas. This approach works well for the majority of scenarios.

However, there are still special cases—such as internal testing or experimental features—where users may want to define a custom placement strategy. Therefore, we propose providing a mechanism that allows users to implement their own ReplicaPlacer.

Previously, KIP-660: Pluggable ReplicaPlacer proposed making the ReplicaPlacer pluggable, but the proposal was rejected. The main concern was that an intelligent placement strategy might require a full view of the cluster, which could be prohibitively expensive in large deployments. Furthermore, such implementations could become overly complex and might introduce latency that impacts the controller thread, thus affecting system stability.

The issue of KAFKA-19507 shows a real-world need for solving custom replica assignment problems. It reinforces the point that different usage scenarios often come with unique assignment needs.

Nevertheless, we can classify ReplicaPlacer as an unstable component. This means the mechanism will be exposed to external users, but without any guarantees of compatibility or long-term stability. The configuration is primarily intended for advanced users and can be useful in internal testing, specialized scenarios, or controlled experiments. By allowing different implementations to be injected, we gain flexibility in the placement logic for testing purposes, while ensuring that the stability of the controller is not compromised.

Public Interfaces

New broker configuration:

New public package:

New public unstable classes in `org.apache.kafka.metadata.placement` package

Proposed Changes

Add a new configuration: replica.placer.class.name. This configuration allows users to define a custom ReplicaPlacer implementation.

public static final String REPLICA_PLACER_CLASS_NAME = "replica.placer.class.name";
public static final String REPLICA_PLACER_CLASS_NAME_DEFAULT = StripedReplicaPlacer.class.getName();     
public static final String REPLICA_PLACER_CLASS_NAME_DOC = "The fully qualified class name of the " +
        "ReplicaPlacer implementation. This plugin is considered unstable, meaning the Kafka " +
        "community does not provide guarantees of compatibility or long-term stability. It is used " +
        "by the broker to determine replica placement when topics or partitions are created. This " +
        "configuration is intended for advanced use cases such as internal testing environments, " +
        "experimental feature development, performance evaluation, or benchmarking. Custom " +
        "implementations should be lightweight and efficient to avoid degrading controller performance" +
        " or impacting overall cluster stability.";

The ReplicaPlacer interface has been updated to extend both Configurable and Closeable to support proper configuration and resource cleanup.

package org.apache.kafka.metadata.placement;

/**
 * The interface which a Kafka replica placement policy must implement.
 */
@InterfaceStability.Unstable
public interface ReplicaPlacer extends Configurable, Closeable {
    /**
     * Create a new replica placement.
     *
     * @param placement     What we're trying to place.
     * @param cluster       A description of the cluster we're trying to place in.
     *
     * @return              A topic assignment.
     *
     * @throws InvalidReplicationFactorException    If too many replicas were requested.
     */
    TopicAssignment place(
        PlacementSpec placement,
        ClusterDescriber cluster
    ) throws InvalidReplicationFactorException;
}

The following classes should also be marked with @InterfaceStability.Unstable.

Also move StripedReplicaPlacer into internal package `org.apache.kafka.metadata.placement.internal`, and add `package-info.java` into `org.apache.kafka.metadata.placement`

/**
 * Provides Replica placer and related classes. 
 */
package org.apache.kafka.metadata.placement;

Compatibility, Deprecation, and Migration Plan

This KIP adds new config which doesn’t break compatibility.

Test Plan

Add new test for new internal config internal.replica.placer.class.name

Rejected Alternatives

KIP-660: Pluggable ReplicaPlacer, but the proposal was ultimately rejected due to the following concerns:

Make replica.placer.class.name as an interanl config

Using Admin API for Post-Creation Reassignment

Users could create topics with default placement and then use kafka-reassign-partitions or Admin API to reassign partitions, but rejected due to the following concerns:

Specifying Assignment During Topic Creation 

Users could use Admin API's createTopics with explicit ReplicaAssignments when creating topics, but rejected due to the following concerns: