Note this was initially erroneously assigned as KIP-178, which was already taken, and has been reassigned KIP-179.
Table of Contents |
---|
Status
Current state: Under Discussion [One of "Under Discussion", "Accepted", "Rejected"]
Discussion thread: here (when initially misnumbered as KIP-178) and here (when assigned KIP-179)here [Change the link from the KIP proposal email archive to your own email thread]
JIRA: here
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation
Describe the problems you are trying to solve.
Firstly, the ReassignPartitionsCommand
(which is used by the kafka-reassign-partitions.sh
tool) talks directly to ZooKeeper. This prevents the tool being used in deployments where only the brokers are exposed to clients (i.e. where the zookeeper servers are intentionally not exposed). In addition, there is a general push to refactor/rewrite/replace tools which need ZooKeeper access with equivalents which use the AdminClient API. Thus it is necessary to change the ReassignPartitionsCommand
so that it no longer talks to ZooKeeper directly, but via an intermediating broker.
Secondly, ReassignPartitionsCommand
currently has no proper facility to report progress of a reassignment; --verify
can be used periodically to check whether the request assignments have been achieved. It would be useful if the tool could report progress better.
Public Interfaces
Briefly list any new interfaces that will be introduced as part of this proposal or any existing interfaces that will be removed or changed. The purpose of this section is to concisely call out the public contract that will come along with this feature.
A public interface is any change to the following:
Binary log format
The network protocol and api behavior
Any class in the public packages under clientsConfiguration, especially client configuration
org/apache/kafka/common/serialization
org/apache/kafka/common
org/apache/kafka/common/errors
org/apache/kafka/clients/producer
org/apache/kafka/clients/consumer (eventually, once stable)
Monitoring
Command line tools and arguments
- Anything else that will likely break existing users in some way when they upgrade
Two new network protocol APIs will be added:
PartitionAssignmentRequest
andPartitionAssignmentResponse
PartitionAssignmentStatusRequest
andPartitionAssignmentStatusResponse
The AdminClient
API will have two new methods added (plus overloads for options):
assignPartitions()
partitionAssignmentStatus()
The options accepted by kafka-reassign-partitions.sh
command will change:
--zookeeper
will be deprecated, with a warning message- a new
--bootstrap-server
option will be added - a new
--progress
action option will be added, with a new--request
option
Reassignment requests will be identified by a monotonic integer (the "request id"), which will be apparent in each of these public interfaces. The intention is to avoid ambiguity in the event that one reassignment is immediately followed by another. Note that this KIP will not add support for concurrent reassignments, but having a request id is compatible with disambiguating concurrent reassignments, should support for them be added at a later date.
Proposed Changes
Describe the new thing you want to do in appropriate detail. This may be fairly extensive and have large subsections of its own. Or it may be a few sentences. Use judgement based on the scope of the change.
kafka-reassign-partitions.sh
and ReassignPartitionsCommand
The --zookeeper
option will be retained and will:
- Cause a deprecation warning to be printed to standard error. The message will say that the
--zookeeper
option will be removed in a future version and that--bootstrap-server
is the replacement option. - Perform the reassignment via ZooKeeper, as currently.
...
Jira server ASF JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key KAFKA-5601 Jira server ASF JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key KAFKA-5561
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation
Describe the problems you are trying to solve.
Firstly, the ReassignPartitionsCommand
(which is used by the kafka-reassign-partitions.sh
tool) talks directly to ZooKeeper. This prevents the tool being used in deployments where only the brokers are exposed to clients (i.e. where the zookeeper servers are intentionally not exposed). In addition, there is a general push to refactor/rewrite/replace tools which need ZooKeeper access with equivalents which use the AdminClient
API. Thus it is necessary to change the ReassignPartitionsCommand
so that it no longer talks to ZooKeeper directly, but via an intermediating broker. Similar work is needed for the kafka-topics.sh
tool (which can also change assignments and numbers of partitions and replicas), so common AdminClient
and protocol APIs are desirable.
Secondly, ReassignPartitionsCommand
currently has no proper facility to report progress of a reassignment; --verify
can be used periodically to check whether the request assignments have been achieved. It would be useful if the tool could report progress better.
Public Interfaces
Briefly list any new interfaces that will be introduced as part of this proposal or any existing interfaces that will be removed or changed. The purpose of this section is to concisely call out the public contract that will come along with this feature.
A public interface is any change to the following:
Binary log format
The network protocol and api behavior
Any class in the public packages under clientsConfiguration, especially client configuration
org/apache/kafka/common/serialization
org/apache/kafka/common
org/apache/kafka/common/errors
org/apache/kafka/clients/producer
org/apache/kafka/clients/consumer (eventually, once stable)
Monitoring
Command line tools and arguments
- Anything else that will likely break existing users in some way when they upgrade
Two new network protocol APIs will be added:
The AdminClient
API will have two new methods added (plus overloads for options):
The options accepted by kafka-reassign-partitions.sh
command will change:
--zookeeper
will be deprecated, with a warning message- a new
--bootstrap-server
option will be added
...
- a new
--progress
action option will be added
Proposed Changes
Describe the new thing you want to do in appropriate detail. This may be fairly extensive and have large subsections of its own. Or it may be a few sentences. Use judgement based on the scope of the change.
kafka-reassign-partitions.sh
and ReassignPartitionsCommand
The --zookeeper
option will be retained and will:
- Cause a deprecation warning to be printed to standard error. The message will say that the
--zookeeper
option will be removed in a future version and that--bootstrap-server
is the replacement option. - Perform the reassignment via ZooKeeper, as currently.
A new --bootstrap-server
option will be added and will:
- Perform the reassignment via the given intermediating broker.
Using both --zookeeper
and --bootstrap-server
in the same command
- Perform the reassignment via the given intermediating broker.
Using both --zookeeper
and --bootstrap-server
in the same command will produce an error message and the tool will exit without doing the intended operation.
It is anticipated that a future version of Kafka would remove support for the --zookeeper
option.
New --progress
and --request
options will be added. These will only be supported when used with --bootstrap-server
. If used with --zookeeper
the command will produce an error message and the tool will exit without doing the intended operation. The existing --execute
option will cause the request id to be printed to the standard output. This can be used as an argument to the --request
option to identify the request that the --progress
pertains to. If --progress
is used without a --request
option present the most recent request will be used. If an invalid --request
option is given the tool will exit with an error message. For example:.
It is anticipated that a future version of Kafka would remove support for the --zookeeper
option.
A new --progress
action option will be added. This will only be supported when used with --bootstrap-server
. If used with --zookeeper
the command will produce an error message and the tool will exit without doing the intended operation. --progress
will report on the synchronisation of each of the partitions and brokers in the reassignment given via the --reassignment-json-file
option
For example:
No Format |
---|
# If the following command is used to start a reassignment
bin/kafka- |
Code Block |
# Start the execution of a reassignment and exit immediately, # printing a reassignment request id to standard output kafka-reassign-partitions.sh --bootstrap-server localhost:12349878 \ --reassignment-json-executefile ... expand-cluster-reassignment.json \ --execute # Assumingthen the abovefollowing command printed 42, the following # will print the progress of # that reassignment, then exit immediately bin/kafka-reassign-partitions.sh --bootstrap-server localhost:12349878 \ --progress reassignment-json-request 42 ... |
Internally, the ReassignPartitionsCommand
will be refactored to support the above changes to the options. An interface will abstract the commands currently issued directly to zookeeper.
There will be an implementation which makes the current calls to ZooKeeper, and another implementation which uses the AdminClient
API described below.
In all other respects, the public API of ReassignPartitionsCommand
will not be changed.
AdminClient
The following methods will be added to AdminClient
to support the ability to reassign partitions:
Code Block |
---|
public AssignPartitionsResult assignPartitions(Map<TopicPartition, List<Integer>>)
public AssignPartitionsResult assignPartitions(Map<TopicPartition, List<Integer>>, AssignPartitionOptions options) |
Where:
file expand-cluster-reassignment.json \
--progress |
That might print something like the following:
No Format |
---|
Topic Partition Broker Status
-------------------------------------
my_topic 0 0 In sync
my_topic 0 |
Code Block |
public class AssignPartitionsResult { // private constructor 1 publicBehind: Map<TopicPartition,10456 KafkaFuture<Void>>messages values() behind asdf 0 public KafkaFuture<Void> all() } |
The following methods will be added to AdminClient to support the progress reporting functionality:
Code Block |
---|
public PartitionAssignmentStatusResult partitionAssignmentStatus(int requestId) 1 Unknown topic publicmy_topic PartitionAssignmentStatusResult partitionAssignmentStatus(int requestId,42 PartitionAssignmentStatusOptions options) |
Where:
Code Block |
---|
public class PartitionAssignmentStatusResult { 1 // private constructor public Map<TopicPartition, KafkaFuture<Void>> values() Unknown partition my_topic 0 public KafkaFuture<Void> all() } public class PartitionAssignmentStatusResult { 42 // private constructor Unknown broker my_topic 1 public Map<PartitionReplica, Progress> all() 0 /** The progressBroker ofdoes movingnot thehost given partition */ public KafkaFuture<Progress> progress(PartitionReplica) /** * Whether the overall reassignment is complete, * that is, if all the values in the map returned by {@link #all()} * have complete() == true */ public KafkaFuture<Boolean> complete() } /** Represents the reassignment of a partition to a broker. */ public class PartitionReplica { public String getTopic() public int getPartition() public int getBroker() // plus override equals() and hashCode() } public class Progress { /** The number of completed units of work. */ long getNumerator() /** The total number of units of work. */ Long getDenominator() /** The error code, if the task ended erroneously. */ Errors getError() } |
The Progress
class presented here can easily be reused in future APIs also dealing with progress of long running jobs. The result of getDenominator()
is nullable because it cannot be expected that all tasks can efficiently compute their total size. The isComplete()
method is provided to determine completeness since getNumerator() == getDenominator()
cannot be used in the event of a null denominator.
In general, the denominator will not be constant between successive requests since new messages will arrive from producers. Indeed it is possible for a partition to receive messages from producers at a faster rate than the syncing replica is catching up, i.e. the percentage completion implied by numerator ÷ denominator can decrease.
For partition reassignment, Progress
has sufficient information for a percent completion calculation, but not any kind of time-to-completion, since the AdminClient doesn't know the time that the initial request was made.
Authorization
With broker-mediated reassignment it becomes possible limit the authority to perform reassignment to something finer-grained than "anyone with access to zookeeper".
The reasons for reassignment are usually operational. For example, migrating partitions to new brokers when expanding the cluster, or attempting to find a more balanced assignment (according to some notion of balance). These are cluster-wide considerations and so authority should be for the reassign operation being performed on the cluster.
Given the standard form for authorization in Kafka
"Principal P is [Allowed/Denied] Operation O From Host H On Resource R"
the authorized operation will be ClusterAction
, on the CLUSTER
resource. So to authorize Alice and Bob to perform rebalancing one might need to configure an ACL like this:
bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181 \
--add --allow-principal User:Bob --allow-principal User:Alice \
--allow-host 198.51.100.0 --allow-host 198.51.100.1 \
--operation ClusterAction --cluster
Network Protocol: AssignPartitionsRequest
and AssignPartitionsResponse
An AssignPartitionsRequest
will initiate the process of partition reassignment
Code Block |
---|
AssignPartitionsRequest => [PartitionAssignment]
PartitionAssignment => Topic Partition Brokers
Topic => string
Partition =>int32
Brokers => [int32] |
Where:
Topic
a topic namePartition
a partition of that topicBrokers
the broker ids which will host the partition
Possible Error Codes:
ClusterAuthorizationFailedCode
(31)PartitionReassignmentInProgress
(new)
As currently, it will not be possible to have multiple reassignments running concurrently, hence the addition of PartitionReassignmentInProgress
It is not necessary to send an AssignPartitionsRequest
to the leader for a given partition. Any broker will do.
The AssignPartitionsResponse
enumerates those topics and partitions in the request, together with any error in initiating reassignment that partition:
Code Block |
---|
AssignPartitionsResponse => RequestId [PartitionAssignmentResult]
RequestId => int32
PartitionAssignmentResult => Topic Partition Broker Error
Topic => string
Partition => int32
Broker => int32
Error => int16 |
Where:
RequestId
is an id which can be used to make in aPartitionAssignmentStatusRequest
Topic
is a topic namePartition
is a partition idBroker
is the brokerError
is an error code for why the initiation of assignment of the given topic to the given broker failed
The anticipated errors in the header are:
INVALID_TOPIC_EXCEPTION
(17) If the topic doesn't existINVALID_PARTITIONS
(37) If the partition doesn't existUNKNOWN_MEMBER_ID
(25) If theBrokers
in theAssignPartitionsRequest
included an unknown broker id
Network Protocol: PartitionAssignmentStatusRequest and PartitionAssignmentStatusResponse
To request progress information about the given request
Code Block |
---|
PartitionAssignmentStatusRequest => RequestId
RequestId => int32 |
Where:
RequestId
is the identifier from a previousAssignPartitionsResponse
Code Block |
---|
PartitionAssignmentStatusResponse => [RequestId] [PartitionReplica Progress]
PartitionReplica => Topic Partition Broker
Topic => string
Partition => int32
Broker => int32
Progress => Error Numerator Denominator
Error => int16
Numerator => int64
Denominator => int64 |
Where:
RequestId
is the identifier from a previous AssignPartitionsResponseTopic
the a topic namePartition
a partition of the topicBroker
the id of the receiving brokerError
is an error code about the transfer of the partition to the given broker:PartitionReassignmentInProgress
(new) if the reassignment is still in progress,None
(0) if the reassignment completed normally,- Any other error if the reassignment completed exceptionally.
Numerator
the number of items that have been processed so farDenominator
the total number of items we expect to process for the request. Or -1 if the total is not known. Never 0.
The anticipated errors in the header are:
INVALID_REQUEST
(42) If the request id is not known
As with the AdminClient API, the Progess
Schema could be reused in other progress reporting APIs.
Implementation
- The mediating broker will write the reassignment JSON to the
/admin/reassign_partitions
znode, as present. The znode version for/admin/reassign_partitions
becomes the request_id. This means that the request id is monotonic increasing, but that request id may not be contiguous. The monotonic increasing property is only required so that when the request id is omitted it is easy to figure out the last request. - Receiving brokers are notified of the change to
/admin/reassign_partitions
(thus obtaining the request id) and start fetching replicas as necessary, as present - As the leader serves fetch requests to followers, at the same time as deciding whether to update the ISR, it will publish their progress as JSON to
/admin/reassign_partitions_status/$request_id
- To obtain progress the mediating broker will read the
/admin/reassign_partitions_status/$request_id
znodes. - When the leader adds a syncing broker to the ISR the progress JSON for that partition in
/admin/reassign_partitions_status/$request_id
will be marked as complete.
this partition |
Internally, the ReassignPartitionsCommand
will be refactored to support the above changes to the options. An interface will abstract the commands currently issued directly to zookeeper.
There will be an implementation which makes the current calls to ZooKeeper, and another implementation which uses the AdminClient
API described below.
In all other respects, the public API of ReassignPartitionsCommand
will not be changed.
AdminClient: alterTopics()
Anchor | ||||
---|---|---|---|---|
|
AdminClient
to support the ability to reassign partitions:Code Block |
---|
/**
* Request alteration of the given topics. The request can change the number of
* partitions, replication factor and/or the partition assignments.
* This can be a long running operation as replicas are migrated between brokers,
* therefore the returned result conveys whether the alteration has been
* started, not that it is complete. Progress information
* can be obtained by calling the lead broker's
* {@link #replicaStatus(Collection)}.
*/
public AlterTopicsResult alterTopics(Collection<AlteredTopic> alteredTopics)
public AlterTopicsResult alterTopics(Collection<AlteredTopic> alteredTopics, AlterTopicsOptions options) |
Where:
Code Block |
---|
public class AlteredTopic {
public AlteredTopic(String name, int numPartitions, int replicationFactor, Map<Integer,List<Integer>> replicasAssignment) {
// ...
}
/** The name of the topic to alter. */
public String name();
/** The new number of partitions, or -1 if the number of partitions should not be changed. */
public int numPartitions();
/** The new replication factor, or -1 if the replication factor should not be changed. */
public short replicationFactor();
/**
* The new assignments of partition to brokers, or the empty map
* if the broker should assign replicas automatically.
*/
Map<Integer,List<Integer>> replicasAssignment();
}
public class AlterTopicsOptions {
public AlterTopicsOptions validateOnly(boolean validateOnly);
public boolean validateOnly();
public AlterTopicsOptions timeoutMs(long timeoutMs);
public long timeoutMs();
}
public class AlterTopicsResult {
// package-access constructor
/** A mapping of the name of a requested topic to the error for that topic. */
Map<String, KafkaFuture<Void>> values();
/** Return a future which succeeds if all the topic alterations were accepted. */
KafkaFuture<Void> all();
} |
AdminClient: replicaStatus()
Anchor | ||||
---|---|---|---|---|
|
AdminClient
to support the progress reporting functionality:Code Block |
---|
/**
* Query the replication status of the given partitions.
*/
public ReplicaStatusResult replicaStatus(Collection<TopicPartition> replicas)
public ReplicaStatusResult replicaStatus(Collection<TopicPartition> replicas, ReplicaStatusOptions options) |
Where:
Code Block |
---|
public class ReplicaStatusOptions {
}
public class ReplicaStatusResult {
public KafkaFuture<Map<TopicPartition, List<ReplicaStatus>>> all()
}
/**
* Represents the replication status of a partition
* on a particular broker.
*/
public class ReplicaStatus {
/** The topic about which this is the status of */
String topic()
/** The partition about which this is the status of */
int partition()
/** The broker about which this is the status of */
int broker()
/**
* The time (as milliseconds since the epoch) that
* this status data was collected. In general this may
* be some time before the replicaStatus() request time.
*/
public long statusTime()
/**
* The number of messages that the replica on this broker is behind
* the leader.
*/
public long lag()
} |
Authorization
With broker-mediated reassignment it becomes possible limit the authority to perform reassignment to something finer-grained than "anyone with access to zookeeper".
The reasons for reassignment are usually operational. For example, migrating partitions to new brokers when expanding the cluster, or attempting to find a more balanced assignment (according to some notion of balance). These are cluster-wide considerations and so authority should be for the reassign operation being performed on the cluster. Therefore alterTopics()
will require ClusterAction
on the CLUSTER
.
replicaStatus()
will require Describe
on the CLUSTER
.
Network Protocol: AlterTopicsRequest
and AlterTopicsResponse
Anchor | ||||
---|---|---|---|---|
|
AlterTopicsRequest
will initiate the process of topic alteration/partition reassignmentNo Format |
---|
AlterTopicsRequest => [alter_topic_requests] validate_only
alter_topic_requests => topic num_partitions replication_factor [partition_assignment]
topic => STRING
num_partitions => INT32
replication_factor => INT16
partition_assignment => partition_id brokers
partition_id => INT32
brokers => [INT32]
validate_only => BOOLEAN
timeout => INT32 |
Where
FIELD | DESCRIPTION |
---|---|
topic | the topic name |
num_partition | the number of partitions. A |
replication_factor | the replication factor. A |
partition_id | the partition id |
brokers | the ids of the assigned brokers for this partition |
validate_only | true to just validate the request, but not actually alter the topics |
timeout | the timeout, in ms, to wait for the topic to be altered. |
An empty partition_assignment
would mean that the broker should calculate a suitable assignment. Such broker calculated assignment is unlikely to be balanced.
It is not necessary to send an AlterTopicsRequest
to the leader for a given partition. Any broker will do.
Anchor | ||||
---|---|---|---|---|
|
AlterTopicsResponse
enumerates those topics in the request, together with any error in initiating alteration:No Format |
---|
AlterTopicsResponse => throttle_time_ms [topic_errors]
throttle_time_ms => INT32
topic_errors => topic error_code error_message
topic => STRING
error_code => INT16
error_message => NULLABLE_STRING |
Where
Field | Description |
---|---|
throttle_time_ms | duration in milliseconds for which the request was throttled |
topic | the topic name |
error_code | the error code for altering this topic |
error_message | detailed error information |
Possible values for error_code
:
CLUSTER_AUTHORIZATION_FAILED
(31) Authorization failedINVALID_TOPIC_EXCEPTION
(17) If the topic doesn't existINVALID_PARTITIONS
(37) If then
um_partitions
was invalidINVALID_REPLICATION_FACTOR
(38) If ther
eplication_factor
was invalidUNKNOWN_MEMBER_ID
(25) If any broker ids in thep
artition_assignment
included an unknown broker idINVALID_REQUEST
(42) If trying to modify the partition assignment and the number of partitions or the partition assignment and the replication factor in the same request. Or if duplicatetopic
s appeared in the request.PARTITION_REASSIGNMENT_IN_PROGRESS
(new)INVALID_REPLICA_ASSIGNMENT
(39) If a partition, replica or broker id in thepartition_assignment
doesn't exist or is incompatible with the requestednum_partitions
and /orreplication_factor
. Theerror_message
would contain further information.NONE
(0) If the request was successful and the alteration/reassignment has been started.
As currently, it will not be possible to have multiple reassignments running concurrently, hence the addition of the PARTITION_REASSIGNMENT_IN_PROGRESS
error code.
Policy
The existing CreateTopicPolicy
can be used to apply a cluster-wide policy on topic configuration at the point of creation via the create.topic.policy.class.name
config property. To avoid an obvious loophole, it is necessary to also be able to apply a policy to topic alteration. Maintaining two separate policies in sync is a burden both in terms of class implementation and configuring the policy. It seems unlikely that many use cases would require a different policy for alteration than creation. On the other hand, just applying the CreateTopicPolicy
to alterations is undesirable because:
- Its name doesn't convey that it would be applied to alterations too
- Its API (specifically its
RequestMetadata
member class) includes topicconfigs
(i.e.Map<String, String>
) which is not part of the API for topic alteration even though it is part of the API for topic creation. - It prevents any use cases which legitimately did need to apply a different policy for alteration than creation.
Finding a balance between compatibility with existing deployments, and not opening the loophole is difficult.
The existing create.topic.policy.class.name
config would continue to work, and would continue to name an implementation of CreateTopicPolicy
. That policy would be applied to alterations automatically. The topic's config would be presented to the validate()
method (via the RequestMetadata
) even though it's not actually part of the AlterTopicsRequest
. The documentation for the interface and config property would be updated.
Network Protocol: ReplicaStatusRequest
and ReplicaStatusResponse
Anchor | ||||
---|---|---|---|---|
|
ReplicaStatusRequest
requests information about the progress of a number of replicas.No Format |
---|
ReplicaStatusRequest => [replica_status_requests]
replica_status_requests => topic partition_id broker
topic => STRING
partition_id => INT32
broker => INT32 |
Where
Field | Description |
---|---|
topic | a topic name |
partition_id | a partition id of this topic |
broker | a follower broker id for this partition |
Anchor | ||||
---|---|---|---|---|
|
No Format |
---|
ReplicaStatusResponse => [replica_status]
replica_status => topic partition_id broker error_code status_time lag
topic => STRING
partition_id => INT32
broker => INT32
error_code => INT16
status_time => INT64
lag => INT64 |
Where
Field | Description |
---|---|
topic | the topic name |
partition_id | the partition id of this topic |
broker | the follower broker id |
error_code | an error code |
status_time | the time the status was current |
lag | the lag (#messages) of this broker, for this partition |
Anticipated errors are:
CLUSTER_AUTHORIZATION_FAILED
(31) Authorization failed. (or the TOPIC?)INVALID_TOPIC_EXCEPTION
(17) The topic is not knownINVALID_PARTITIONS
(37) Thepartion
_id
of the given topic is not validUNKNOWN_MEMBER_ID
(25) The givenbroker
id is not known.UNKNOWN_TOPIC_OR_PARTITION
(3) The givenbroker
is not a follower for the partition identified bytopic
,partition
.NONE
(0) if the status request completed normally,
Implementation
The AdminClient.replicaStatus()
will make the underlying ReplicaStatusRequest
to the leader for the given partition. This saves the need for every broker (because any broker could be the --bootstrap-server
) to have knowledge of the replication status of every replica, which would be inefficient in network IO and/or memory useIt will be necessary to delete the /admin/reassign_partitions_status/$request_id
znodes to prevent the /admin/reassign_partitions_status
subtree growing arbitrarily. At the same time the leader publishes status to the current /admin/reassign_partitions_status/$request_id
it will remove znodes which had completed a given amount of time ago. This will be configurable via the broker property reassign.partitions.status.min.retention.ms
which will default to 1 day.
Compatibility, Deprecation, and Migration Plan
- What impact (if any) will there be on existing users?
- If we are changing behavior how will we phase out the older behavior?
- If we need special migration tools, describe them here.
- When will we remove the existing behavior?
Existing users of the kafka-reassign-partitions.sh
will receive a deprecation warning when they use the --zookeeper
option. The option will be removed in a future version of Kafka. If this KIP is introduced in version 1.0.0 the removal could happen in 2.0.0.
Rejected Alternatives
If there are alternative ways of accomplishing the same thing, what were they? The purpose of this section is to motivate why the design is the way it is and not some other way.
...
Another alternative is to do exactly this KIP, but without the deprecation of --zookeeper
. That would have a higher long term maintenance burden, and would prevent any future plans to, for example, provide alternative cluster technologies than ZooKeeper.