Page History

...

// Optional values
--recovery-duration-ms <integer>			A broker is required to respond with log information within this duration for it to be considered as a candidate for election.
											Defaults to 30_000 (30 seconds) and represents time in milliseconds.
--recovery-election-attempts <integer>      Number of retries of transient failures allowed for leader election requests.
                                            Defaults to 3. 

// At least 1 of these is required.

--show-replica-info							Print a table showing replica info of targeted partitions. Can be used with any other arguments.

--manual-recovery-output-file <string>		File path of a new file which will be created. The generated file will be capable of being an input to kafka-elect-leaders.sh.
											It will contain topic-partitions and their designated leaders.
                                            {
												"partitions": [
													{"topic": "foo", "partition": 1, "designatedLeader": 0},
	                                        	 	{"topic": "foobar", "partition": 2, "designatedLeader": 1}
												]
                                        	}
											Mutually exclusive with the --automated-recovery flag.

--automated-recovery						Automatically elect leaders with longest apparent logs by attempting designated leader elections. 


// One of these two are required.                    
--path-to-json-file <String>				Path to a JSON file containing a list of topic-partitions to attempt unclean recovery elections on. 
											Mutually exclusive with --all-offline-partitions.
											Example:
											{
												"partitions": [
													{
														topic: "foo",
														partitions: [0, 3, 5]
													},
													{
														topic: "bar",
														partitions: [0, 1, 4]
													}
												]
											}

--all-offline-partitions			Perform unclean recovery on all detected offline partitions.

...

Limit: 1000 partitions per request. If more than 1000 partitions are included, only the first 1000 will be served. Others will be returned with REQUEST_LIMIT_REACHED.

{
  "apiKey": XX,
  "type": "request",
  "listeners": ["broker", "controller"],
  "name": "ElectLeadersRequest",
  "validVersions": "0-3",
  "flexibleVersions": "2+",
  "fields": [
  ...
  { "name": "TopicPartitions", "type": "[]TopicPartitions", "versions": "0+", "nullableVersions": "0+",
    "about": "The topic partitions to elect leaders.",
    "fields": [
    ...

// New fields begin. The same level with the Partitions
     { "name": "DesignatedLeaders", "type": "[]int32", "versions": "3+", "nullableVersions": "3+",
       "about": "The designated leaders. The entry should match with the entry in Partitions by the index." },
     },
// New fields end.

  ] },
  { "name": "TimeoutMs", "type": "int32", "versions": "0+", "default": "60000",
    "about": "The time in ms to wait for the election to complete." }
  ] 
}

...

Limit: 1000 partitions per request. If more than 1000 partitions are included, only the first 1000 will be served. Others will be returned with REQUEST_LIMIT_REACHED.

{
  "apiKey":XX,
  "type": "request",
  "listeners": ["broker"],
  "name": "GetReplicaLogInfoRequest",
  "validVersions": "0",
  "flexibleVersions": "0+",
  "fields": [
    { "name": "TopicPartitions", "type": "[]TopicPartitions", "versions": "0+", "nullableVersions": "0+",
      "about": "The topic partitions to query the log info for.",
      "fields": [
        { "name": "TopicId", "type": "uuid", "versions": "0+", "about": "The unique topic ID"},
        { "name": "Partitions", "type": "[]int32", "versions": "0+",
          "about": "The partitions of this topic whose leader should be elected." }
      ]}
  ]
}

...

For the tool to work as intended, GetReplicaLogInfo RPC and designated leadership extension to ElectLeaders RPC are needed. While it is not explicitly specified in KIP-966, designated leadership must have specific safety properties for the tool to operate safely. Designated leader elections should only be allowed if the partition is offline (IE has no leader). A broker may only be designated a leader of a partition if it is in the replica set, is unfenced and has an online replica. This behaviour is consistent with the prerequisites for the existing "unclean" leadership elections. GetReplicaLogInfo requests ask the broker for the length of its replica of a given partition. Note GetReplicaLogInfo in this KIP is slightly different from the one specified in KIP-966.  

...

kafka-unclean-recovery.sh will attempt a "best-effort" approach and try to complete as many elections as it can within a time limit (or until the operator provides a SIGTERM). It should be expected that not all partitions in the input set will be "recovered" successfully. Possible failure modes are that either ElectLeadersRequest will fail or no successful GetReplicaLogInfoResponse was received in time for a given partition. Since designated leadership elections are idempotent (once a leader is elected, it is not possible to use designated leadership elections again until partition is again offline) operators can confidently run the tool multiple times. A zero exit status will be reported only if all elections for the input set are successful. Any failed elections will be logged to stderr. 

There are some caveats to using a command line tool for this system.

...