Master KIP
KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum (Accepted)
Status
Current state: accepted
...
JIRA:
Jira | ||||||
---|---|---|---|---|---|---|
|
Release: controller-side changes in 2.4, command line changes in 2.6
Motivation
Currently, users initiate replica reassignment by writing directly to a ZooKeeper node named /admin/reassign_partitions.
...
Code Block | ||
---|---|---|
| ||
/** * Change the reassignments for one or more partitions. * Providing an empty Optional (e.g via {@link Optional#empty()}) will <bold>cancel</bold> the reassignment for the associated partition. * * @param reassignments The reassignments to add, modify, or remove. * @param options The options to use. * @return The result. */ public AlterPartitionReassignmentsResult alterPartitionReassignments( Map<TopicPartition, Optional<NewPartitionReassignment>> reassignments, AlterPartitionReassignmentsOptions options); /** * A new partition reassignment, which can be applied via {@link AdminClient#alterPartitionReassignments(Map)}. */ public class NewPartitionReassignment { private final List<Integer> targetReplicas; /** * @throws IllegalArgumentException if no replicas public static Optional<NewPartitionReassignment> ofare supplied */ public NewPartitionReassignment(List<Integer> replicastargetReplicas) {} public List<Integer> targetReplicas() } class AlterPartitionAssignmentsResult { Map<TopicPartition, KafkaFuture<Void>> values() Future<Void> all(); // Throws an exception if any reassignment was rejected } class AlterPartitionAssignmentsOptions extends AbstractOptions<> { // contains timeoutMs } /** * List some of the current partition reassignments. * * @param partitions The partitions to show reassignments for. Must be non-null. * @param options The options to use. */ public ListPartitionReassignmentsResult listPartitionReassignments( Set<TopicPartition> partitions, ListPartitionReassignmentsOptions options); /** * List all of the current partition reassignments. * * @param options The options to use. */ public ListPartitionReassignmentsResult listPartitionReassignments( ListPartitionReassignmentsOptions options); abstract ListPartitionReassignmentsResult listPartitionReassignments(Optional<Set<TopicPartition>> partitions, ListPartitionReassignmentsOptions options) class ListPartitionReassignmentsOptions extends AbstractOptions<> { // contains timeoutMs } class ListPartitionReassignmentsResult { private final KafkaFuture<Map<TopicPartition, PartitionReassignment>> reassignments; } /** * A partition reassignment, which has been listed via {@link AdminClient#listPartitionReassignments()}. */ public class PartitionReassignment { /** * The brokers which this partition currently resides on. */ private final List<Integer> replicas; /** * The brokers that we are adding this partition to as part of a reassignment. */ private final List<Integer> addingReplicas; /** * The brokers that we are removing this partition from as part of a reassignment. */ private final List<Integer> removingReplicas; } |
...
kafka-reassign-partitions.sh is a script to create, inspect, or modify partition reassignments.
Deprecation of --zookeeper
We will remove deprecate the --zookeeper option from in this script. New operations introduced by this KIP (cancellation, list) will not be supported by the ZK flag.
All old operations and new will be supported Instead, all operations will be performed through the admin client. This will be initialized by the --bootstrap-server flag, which already exists.
The --list flag
There will be a new flag, --list, which will list all of the current replica reassignments that are going on in the cluster. They will be output in the same JSON format that is accepted by --reassignment-json-fileFor each reassignment, we will list the replica set, the adding replicas, and the removing replicas. If there are no ongoing replica reassignments, an empty JSON object "No partition reassignments found." will be printed.
...
The --additional flag
Like the underlying KIP-455 API, the reassign-partitions tool will now reassign-partitions tool will work in an incremental fashion – any . Any new reassignment will be added on top of the current ones. It will continue to output a rollback JSON as part of any reassignment in order to let the user revert the changes.
However, to avoid confusing users, the tool will decline to create a new reassignment if there is already one in progress. Users can choose to create additional reassignments even if there is one in progress by specifying the new --additional flag. This flag is only supported when --bootstrap-server is provided.
There is also one slight behavior change to --execute that is worth noting here. Previously, when --execute declined to create a new reassignment because one was already in progress, it would still apply the provided throttle, if any. In a world of concurrent reassignments, this behavior no longer makes sense, and has been removed. (However, users can still modify the throttle by simply supplying the --additional flag)
Cancellation with the tool
Additionally, we will support a --cancelAdditionally, we will support a --create-cancellation-plan flag. This flag will create a JSON file that, if applied, will cancel all on-going partition reassignments.
...