Page History

...

When creating topics or partitions, the Kafka controller has to pick brokers to host the new partitions. The current placement logic is based on a round robin algorithm and supports rack awareness. While this works relatively well in many scenarios, in a few cases the placements it generates are not optimal because it's not aware of the state of the clusters. Many cluster administrators then rely on tools like Cruise Control to move partitions to better brokers. This process is expensive as often data has to be copied between brokers.

It would be desirable to allow custom logic for the placer to leverage the state of the cluster and minimize the number of partition reassignments necessary. It would enable administrators to build assignment rules for their clusters and minimize the number of partition reassignments necessary.

Some scenarios that could benefit greatly from this feature:

...