Status

Current state: UNDER DISCUSSION

Discussion thread: <link to mailing list DISCUSS thread>

JIRA: KAFKA-5319

Released: 

Motivation

I think kafka need to add a tool to make amounts of replicas and leaders on every brokers balanced.Current kafka server does not provide a function to make cluster balance.When we use console command to create a topic without 'replicaAssignmentOpt' ,kafka server   assigns replicas  by a "replicaAssignment" function . The function spread the replicas evenly among brokers to make cluster balanced.It can not make sure the cluster is balanced.Even though it is balanced,the balanced state will be destroyed.Example:

1.Adding some new broker to current kafka cluster.There is nothing in a new broker.The amounts of replicas and leaders in new brokers are always fewer than old ones. 

2.Using ReassignPartitionsCommand will change the replica and leader balance of cluster.

    Example.  reassignment-json-file as :

    { "partitions":

         [{"topic": "topic-test",

           "partition": 0,

           "replicas": [1,2,3] },

         {"topic": "topic-test",

           "partition": 1,

           "replicas": [1,2,3] },

          {"topic": "topic-test",

           "partition": 2,

           "replicas": [1,2,3] }

        ],

         "version":1

      }

Some one want to assign replicas of 3 partitions  to broker 1,2,3.The one was accustomed to write AR  in ascending order.Kafka server choose a  PreferredReplicaLeader which be put at the first position of  the AR to make leader balance .If all leaders are the PreferredReplicaLeader of AR ,kafka server thinks leaders in cluster is balanced.In above case,kafka server thinks it is balance when all leaders are replica 1.In fact the leader balance is not good at that time.

So it is necessary to add to add a tool to deal with replica and leader balance. 

Proposed Changes

We use an algorithm to get a balanced assignment of replicas or leaders,and call ReassignPartitionsCommand to let kafka sever move replicas or change the position of replicas in ARs. 

We designed 3 command of this tools:

1. show-cluster-state         This command print out current cluster balance state.We can use it to learn whether or not the cluster need                             to be balanced

2. leader-balance             This command let amounts of leaders of every brokers  balanced

3. replica-balance            This command let amounts of replicas of every brokers balanced

    

Public Interfaces

Compatibility, Deprecation, and Migration Plan

Rejected Alternatives

We can also use  ReassignPartitionsCommand to move replicas to make replica balance and change the first replica in ARs to make leader balance by manual operation.However it will become more and more difficult with topic and broker increasing.