...

Current state: "Under Discussion"

Discussion thread: https://lists.apache.org/thread.html/r27728a3b265d9ca26b2ef3672ad632e1a5b8da264ba1897cb01f0cca%40%3Cdev.kafka.apache.org%3Ehere [Change the link from the KIP proposal email archive to your own email thread]

JIRA: 

...

We hope to add an interface to TopicCommand to directly obtain the list of partitions whose leader is in a non-preferred replica.

One of the things a user might do after running this command would be to use `kafka-preferred-replica-election.sh` to try electing the preferred leader for (some of) those partitions.

kafka-preferred-replica-election.sh and kafka-leader-election.sh tools  reads a JSON file containing the partitions which should have the preferred leader elected. 

Therefore, we hope to be able to output json format.

Public Interfaces

kafka-topic.sh:

The topic command will accept the --non-preferred-leader or-under-preferred-replica-partitions option when using the --bootstrap-server option.-non-preferred-leader-json options when using the --bootstrap-server option.

In order to reflect that the non-preferred replica was elected as the leader of the partition, I set auto.leader.rebalance.enable=false

1. --non-preferred-leader for the more human readable output,  one example of use:

kafka-topics.sh --bootstrap-server localhost:9092 --describe --non-preferred-leader

Topic: test-04 Partition: 0 Leader: 1002 Replicas: 1001,1005,1002 Isr: 1002,1005,1001
Topic: test-04 Partition: 1 Leader: 1006 Replicas: 1005,1006,1004 Isr: 1006,1005,1004
Topic: zzt Partition: 0 Leader: 1006 Replicas: 1001,1006,1002 Isr: 1006,1002,1001
Topic: zzt Partition: 1 Leader: 1003 Replicas: 1002,1001,1003 Isr: 1003,1001,1002
Topic: zzt Partition: 3 Leader: 1003 Replicas: 1004,1003,1005 Isr: 1003,1005,1004
Topic: zzt Partition: 4 Leader: 1006 Replicas: 1005,1004,1006 Isr: 1006,1005,1004
Topic: zzt Partition: 6 Leader: 1003 Replicas: 1001,1002,1003 Isr: 1003,1001,1002
Topic: zzt Partition: 7 Leader: 1003 Replicas: 1002,1003,1004 Isr: 1003,1002,1004

...

2. --non-preferred-leader-jsonfor the JSON output,  one example of use:

kafka-topics.sh --bootstrap-server localhost:9092 --describe --non-preferred-leader-json

{"partitions":[{"topic":"__consumer_offsets","partition":42},{"topic":"zzt","partition":16},{"topic":"__consumer_offsets","partition":1},{"topic":"__consumer_offsets","partition":36},{"topic":"__consumer_offsets","partition":48},{"topic":"__consumer_offsets","partition":12},{"topic":"__consumer_offsets","partition":30},{"topic":"__consumer_offsets","partition":18},{"topic":"__consumer_offsets","partition":5},{"topic":"zzt","partition":8},{"topic":"__consumer_offsets","partition":10},{"topic":"__consumer_offsets","partition":24},{"topic":"test-04","partition":0},{"topic":"zzt","partition":4},{"topic":"__consumer_offsets","partition":34},{"topic":"__consumer_offsets","partition":14},{"topic":"__consumer_offsets","partition":37},{"topic":"__consumer_offsets","partition":31},{"topic":"zzt","partition":1},{"topic":"__consumer_offsets","partition":41},{"topic":"zzt","partition":19},{"topic":"__consumer_offsets","partition":2},{"topic":"__consumer_offsets","partition":9},{"topic":"__consumer_offsets","partition":46},{"topic":"__consumer_offsets","partition":33},{"topic":"__consumer_offsets","partition":22},{"topic":"__consumer_offsets","partition":4},{"topic":"zzt","partition":9},{"topic":"__consumer_offsets","partition":21},{"topic":"__consumer_offsets","partition":49},{"topic":"__consumer_offsets","partition":47},{"topic":"test-04","partition":1},{"topic":"__consumer_offsets","partition":25},{"topic":"zzt","partition":5},{"topic":"__consumer_offsets","partition":0},{"topic":"__consumer_offsets","partition":15},{"topic":"zzt","partition":0},{"topic":"__consumer_offsets","partition":28},{"topic":"zzt","partition":2},{"topic":"__consumer_offsets","partition":32},{"topic":"__consumer_offsets","partition":44},{"topic":"zzt","partition":17},{"topic":"__consumer_offsets","partition":20},{"topic":"__consumer_offsets","partition":7},{"topic":"__consumer_offsets","partition":3},{"topic":"__consumer_offsets","partition":16},{"topic":"zzt","partition":14},{"topic":"zzt","partition":3},{"topic":"__consumer_offsets","partition":35},{"topic":"__consumer_offsets","partition":29},{"topic":"zzt","partition":18},{"topic":"zzt","partition":7},{"topic":"zzt","partition":13},{"topic":"__consumer_offsets","partition":11},{"topic":"__consumer_offsets","partition":17},{"topic":"__consumer_offsets","partition":39},{"topic":"__consumer_offsets","partition":43},{"topic":"__consumer_offsets","partition":23},{"topic":"__consumer_offsets","partition":38},{"topic":"__consumer_offsets","partition":26},{"topic":"zzt","partition":15},{"topic":"zzt","partition":10},{"topic":"__consumer_offsets","partition":19},{"topic":"__consumer_offsets","partition":45},{"topic":"__consumer_offsets","partition":40},{"topic":"zzt","partition":12},{"topic":"__consumer_offsets","partition":8},{"topic":"__consumer_offsets","partition":6},{"topic":"zzt","partition":11},{"topic":"zzt","partition":6},{"topic":"__consumer_offsets","partition":27},{"topic":"__consumer_offsets","partition":13}]}

3. We can direct the output of example two to a json file named preferred.json, Then use the kafka-leader-election.sh tool to trigger the preferred replica election.

kafka-leader-election.sh --bootstrap-server localhost:9092 --election-type PREFERRED --path-to-json-file preferred.json

Successfully completed leader election (PREFERRED) for partitions __consumer_offsets-46, zzt-19, __consumer_offsets-44, zzt-15, test-04-1, zzt-13, __consumer_offsets-19, zzt-9, zzt-7, __consumer_offsets-32, zzt-3, __consumer_offsets-28, zzt-1, test-04-0, __consumer_offsets-26, __consumer_offsets-7, __consumer_offsets-40, __consumer_offsets-5, __consumer_offsets-38, __consumer_offsets-36, __consumer_offsets-1, __consumer_offsets-16, zzt-18, __consumer_offsets-14, zzt-16, zzt-14, __consumer_offsets-10, zzt-12, __consumer_offsets-24, zzt-10, __consumer_offsets-22, __consumer_offsets-20, __consumer_offsets-49, zzt-6, __consumer_offsets-31, zzt-4, zzt-0, __consumer_offsets-8, __consumer_offsets-37, __consumer_offsets-6, __consumer_offsets-35, __consumer_offsets-2

Proposed Changes

Make some changes in TopicCommand.scala

    def isUnderPreferredReplicaPartitionsisNonPreferredLeader: Boolean = {
      hasLeader && !info.leader.equals(info.replicas.asScala.head)
    }

    private def shouldPrintUnderPreferredReplicaPartitionsshouldPrintNonPreferredLeader(partitionDescription: PartitionDescription): Boolean = {
      (opts.reportNonPreferredLeader || opts.reportUnderPreferredReplicaPartitionsreportNonPreferredLeaderJson) && partitionDescription.isUnderPreferredReplicaPartitionsisNonPreferredLeader
    }

    private val reportUnderPreferredReplicaPartitionsOptreportNonPreferredLeaderOpt = parser.accepts("undernon-preferred-leader",
      "if set when describing topics, only show partitions whose leader is not equal to the first replica in the replica-partitions list. Not supported with the --zookeeper option.")
    private val reportNonPreferredLeaderJsonOpt = parser.accepts("non-preferred-leader-json",
      "if set when describing topics, only show partitions whose leader is not equal to the first replica in the replica list and output as json format. Not supported with the --zookeeper option.")

    private val allReplicationReportOpts = Set(reportUnderReplicatedPartitionsOpt, reportUnderMinIsrPartitionsOpt, reportAtMinIsrPartitionsOpt, reportUnavailablePartitionsOpt, reportNonPreferredLeaderOpt, reportUnderPreferredReplicaPartitionsOptreportNonPreferredLeaderJsonOpt)

    def reportNonPreferredLeader: Boolean = has(reportNonPreferredLeaderOpt)
    def reportUnderPreferredReplicaPartitionsreportNonPreferredLeaderJson: Boolean = has(reportUnderPreferredReplicaPartitionsOpt)(reportNonPreferredLeaderJsonOpt)

    private def formatAsJson(nonPreferredLeader: Set[TopicPartition]): String = {
      Json.encodeAsString(Map(
        "partitions" -> nonPreferredLeader.map { tp =>
          Map(
            "topic" -> tp.topic(),
            "partition" -> tp.partition()
          ).asJava
        }.asJava
      ).asJava)
    }

	# Some other minor changes omitted

...

The new option has no effect on existing usage.

Rejected Alternatives

1. Do not use --under-preferred-replica-partitions option, a better name for the option would be --non-preferred-leader rather than --under-preferred-replica-partitions.

2.Because the other outputs from kafka-topics.sh don't naturally get used as inputs to the other tools, so such as `--output=json`  top level option would not be a good fit.