THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
| Code Block |
|---|
{
"partitions":
[
{"topic": "topic1", "partition": "0"},
{"topic": "topic1", "partition": "1"},
{"topic": "topic1", "partition": "2"},
{"topic": "topic2", "partition": "0"},
{"topic": "topic2", "partition": "1"}
]
}
|
FAQ
What happens if the preferred replica is not in the ISR?
...
The goal of this tool is similar to the Referred Preferred Replica Leader Election Tool as to achieve load balance across brokers. But instead of only electing a new leader from the assigned replicas of a partition, this tool allows to change the assigned replicas of partitions – remember that followers also need to fetch from leaders in order to keep in sync, hence sometime only balance the leadership load is not enough.
...
Note that the tool only updates the zookeeper path and exits. The controller reassign the replicas for the partitions asynchronously.
This tool is only available in the 0.8 branch for now.
How to use the tool?
WARNING: The tool was released in beta in 0.8 and has some bugs that can render the topic unusable. The tool is known to be stable in 0.8.1.
| Code Block |
|---|
bin/kafka-reassign-partitions.sh Option Description ------ ----------- --brokerbootstrap-listserver <brokerlist> <String: Server(s) the server(s) to use for The list of brokers to whichuse the for bootstrapping> bootstrapping. REQUIRED if an partitions need to be reassigned in absolution path of the log directory the form "0,1,2". This is required is specified for any replica in the for automatic topic reassignment. --execute [execute] reassignment json file --broker-list <String: brokerlist> This option does the actualThe list of brokers to which the partitions reassignment.need Byto default,be thereassigned toolin does a dry run --manual-assignment-json-file <manual The JSON file with the list of manual assignment json file path>the form "0,1,2". This is required reassignmentsThis option or topics- if --topics-to-move-json-file is used to generate reassignment configuration --disable-rack-aware Disable rack aware replica assignment --execute Kick off the reassignment as specified by the --reassignment-json-file option. --generate Generate a candidate partition reassignment configuration. Note that this only generates a candidate assignment, it does not execute it. --reassignment-json-file <String: The JSON file with the partition manual assignment json file path> reassignment configurationThe format to use is - {"partitions": [{"topic": "foo", "partition": 1, "replicas": [1,2,3], "log_dirs": ["dir1","dir2","dir3"] }], "version":1 } Note that "log_dirs" is optional. When it is specified, its length must equal the length of the replicas list. The value in this list can be either "any" or the absolution path of the log directory on the broker. If absolute log directory path is specified, it is currently required that the replica has not already been created on that broker. The replica will then be created in the specified log directory on the broker later. --throttle <Long: throttle> The movement of partitions will be throttled to this value (bytes/sec). Rerunning with this option, whilst a rebalance is in progress, will alter the throttle value. The throttle rate should be at least 1 KB/s. (default: -1) --timeout <Long: timeout> The maximum time in ms allowed to wait for partition reassignment execution to be successfully initiated (default: 10000) --topics-to-move-json-file needs to be<String: Generate a reassignment configuration topics to reassign json file path> to move the partitions of the specified. Thetopics format to usethe islist -of {"partitions": brokers specified by the --broker- [{"topic": "foo", list option. The format to use is - {"partitiontopics": 1, "replicas": [1,2,3] [{"topic": "foo"},{"topic": "foo1"}], "version":1 } --topics-to-move-json-file <topics toverify The JSON file with the list of topics reassign json file path> Verify if to reassign.This option or manual- the reassignment completed as specified by the assignment--json-file needs to be reassignment- json-file option. If there is a specified. The format to use is - throttle engaged for the replicas {"topics": specified, and the rebalance has [{"topic": "foo"},{"topic": "foo1"}], completed, the throttle will "version":1be }removed --zookeeper <urls> <String: urls> REQUIRED: The connection string for the zookeeper connection in the form host:port. Multiple URLS can be given to allow fail-over. |
...
The partition reassignment tool can be used to expand an existing 0.8 Kafka cluster. Cluster expansion involves including brokers with new broker ids in a Kafka 08 cluster. Typically, when you add new brokers to a cluster, they will not receive any data from existing topics until this tool is run to assign existing topics/partitions to the new brokers. The tool allows 2 options to make it easier to move some topics in bulk to the new brokers. These 2 options are a) topics to move b) list of newly added brokers. Using these 2 options, the tool automatically figures out the placements of partitions for the topics on the new brokers and generates new JSON data which can be used in the next step (with the --reassignment-json-file option) to execute the move.
The following example moves 2 topics (foo1, foo2) to newly added brokers in a cluster (5,6,7)
| Code Block |
|---|
nnarkhed$$ ./bin/kafka-reassign-partitions.sh --topics-to-move-json-file topics-to-move.json --broker-list "5,6,7" --generate --executezookeeper localhost:2181 nnarkhed$$ cat topics-to-move.json {"topics": [{"topic": "foo1"},{"topic": "foo2"}], "version":1 } |
...
The following example moves 1 partition (foo-1) from replicas 1,2,3 to 1,2,4
| Code Block |
|---|
nnarkhed$$ ./bin/kafka-reassign-partitions.sh --manual-assignmentreassignment-json-file partitions-to-move.json --execute nnarkhed$$ cat partitions-to-move.json {"partitions": [{"topic": "foo", "partition": 1, "replicas": [1,2,4] }], }], "version":1 } |
5. StateChangeLogMerger Tool
...