THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
| Code Block |
|---|
bin/kafka-reassign-partitions.sh
Option Description
------ -----------
--broker-list <brokerlist> The list of brokers to which the
partitions need to be reassigned in
the form "0,1,2". This is required
for automatic topic reassignment.
--execute [execute] This option does the actual
reassignment. By default, the tool
does a dry run
--manual-assignment-json-file <manual The JSON file with the list of manual
assignment json file path> reassignmentsThis option or topics-
to-move-json-file needs to be
specified. The format to use is -
{"partitions":
[{"topic": "foo",
"partition": 1,
"replicas": [1,2,3] }],
"version":1
}
--topics-to-move-json-file <topics to The JSON file with the list of topics
reassign json file path> to reassign.This option or manual-
assignment-json-file needs to be
specified. The format to use is -
{"topics":
[{"topic": "foo"},{"topic": "foo1"}],
"version":1
}
--zookeeper <urls> REQUIRED: The connection string for
the zookeeper connection in the form
host:port. Multiple URLS can be
given to allow fail-over.
|
Please note that by default the script runs in a dry-run mode and does not initiate the partition movement. Only when the --execute option is specified, the tool proceeds to start the partition movement
Cluster Expansion
The partition reassignment tool can be used to expand an existing 0.8 cluster. Cluster expansion involves including brokers with new broker ids in a Kafka 08 cluster. Typically, when you add new brokers to a cluster, they will not receive any data from existing topics until this tool is run to assign existing topics/partitions to the new brokers. The tool allows 2 options to make it easier to move some topics in bulk to the new brokers. These 2 options are a) topics to move b) list of newly added brokers. Using these 2 options, the tool automatically figures out the placements of partitions for the topics on the new brokers.
The following example moves 2 topics (foo1, foo2) to newly added brokers in a cluster (5,6,7)
| Code Block |
|---|
nnarkhed$ ./bin/kafka-reassign-partitions.sh --topics-to-move-json-file topics-to-move.json --broker-list "5,6,7" --execute
nnarkhed$ cat topics-to-move.json
{"topics":
[{"topic": "foo1"},{"topic": "foo2"}],
"version":1
}
|
Selectively moving some partitions to a broker
The partition movement tool can also be moved to selectively move some replicas for certain partitions over to a particular broker. Typically, if you end up with an unbalanced cluster, you can use the tool in this mode to selectively move partitions around. In this mode, the tool takes a single file which has a list of partitions to move and the replicas that each of those partitions should be assigned to.
The following example moves 1 partition (foo-1) from replicas 1,2,3 to 1,2,4
| Code Block |
|---|
nnarkhed$ ./bin/kafka-reassign-partitions.sh --manual-assignment-json-file partitions-to-move.json --execute
nnarkhed$ cat partitions-to-move.json
{"partitions":
[{"topic": "foo",
"partition": 1,
"replicas": [1,2,4] }],
}],
"version":1
}
|
7. StateChangeLogMerger Tool
...
| Code Block |
|---|
nnarkhed-mn:kafka-git-idea nnarkhed$ ./bin/kafka-run-class.sh kafka.tools.StateChangeLogMerger Provide arguments to exactly one of the two options "[logs]" or "[logs-regex]" Option Description ------ ----------- --end-time <end timestamp in the The latest timestamp of state change state change format java.text. log entries to be merged (default: SimpleDateFormat@f17a63e7> 9999-12-31 23:59:59,999) --logs <file1,file2,...> Comma separated list of state change logs or a regex for the log file names --logs-regex <for example: /tmp/state- Regex to match the state change log change.log*> files to be merged files to be merged --partitions <0,1,2,...> Comma separated list of partition ids whose state change logs should be merged --start-time <start timestamp in the The earliest timestamp of state change format java.text. log entries to be merged (default: SimpleDateFormat@f17a63e7> 0000-00-00 00:00:00,000) --topic <topic> The topic whose state change logs should be merged |