THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
Status
Current state: Under Discussion
Discussion thread:
JIRA: KAFKA-8330
Released:
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation
A related work is KIP-73 which added quotas for replication but it doesn't separate normal replication traffic from reassignment. So a user is able to specify the partition and the throttle rate but it will be applied to both ISR and non-ISR traffic. This is undesirable because if a node that is being throttled falls out of ISR it would further prevent it from catching up. KIP-455 will make brokers aware of pending reassignment and thus we'd be able to separate these two kinds of replication. Moreover we won't have to manually specify a list of replicas to throttle because the broker would be able to figure out which partitions needed to be throttled based on the LeaderAndIsr request.
As part of this KIP we plan to add a broker level dynamic config that defines a throttle rate of the reassignment related replication of the followers in bytes/seconds. As a consequence replication will take more time but we would have the benefit of a stable cluster.
Goals
- Reassignment traffic shouldn't have an effect on other replication or client traffic so they'll get guaranteed throughput.
- This throughput limit should be configured dynamically by the administrator so that an ongoing reassignment's throttle could be controlled on the fly.
- The quota should be able to be upgraded easily programmatically so management systems built on Kafka should be able to expose it easily.
Public Interfaces
- Extra configs: two extra config will be added, called
leader.reassignment.throttled.rateandfollower.reassignment.throttled.rate. Please see proposed changes for more info. - Disabled configs:
leader.replication.throttled.replicasandfollower.replication.throttled.replicaswould be disabled after this change. - Metrics:
- Count of partitions being reassigned per broker
- Read and write rate for partition reassignment per broker in bytes/second
Proposed Changes
Behavior Change of Existing Configs
The primary targets are leader.replication.throttled.replicas and follower.replication.throttled.replicas. These will be rendered obsolete as the replicas to throttle would be computed from the new LeaderAndIsr targetReplicas field that KIP-455 introduces. The reasoning for computing these automatically is makes the administrator's life easier by eliminating some manual steps and the possibility of misconfiguration. Also by disabling this config we won't allow throttling non-reassignment traffic because ISR replicas should have the bandwidth to stay in-sync and non-ISR replicas should also have the bandwidth to catch up. Therefore these configs won't have any effect after this KIP.
Secondly the behavior of leader.replication.throttled.rate and follower.replication.throttled.rate will change slightly as a consequence of the above change. They will still mean leader and follower throttle rate but in this case will only be applied to reassignment and not to replication in general.
New Configs
We will introduce two new configs in order to eventually replace *.replication.throttled.rate. The reason for this is that their name don't express their functionality anymore and therefore should be replaced. From the user's perspective configuring *.replication.throttled.rate or the new configs would be equivalent but in the first case they would got a warning logged.
The new configs are:
| Config name | Type | Default | Valid values | Importance | Dynamic update mode |
|---|---|---|---|---|---|
| leader.reassignment.throttled.rate | Long | Long.MAX_VALUE | [1,...] | medium | per-broker |
| follower.reassignment.throttled.rate | Long | Long.MAX_VALUE | [1,...] | medium | per-broker |
Quota calculation
The quota calculation method that is introduced in KIP-73 wouldn't change in principal but we will only apply it to the calculated (reassigned) replicas accordingly.
Updating quotas
At this point this KIP doesn't aim to add new AdminClient RPC call as the config value can be changed by the IncrementalAlterConfigs API.
Metrics
There will be couple of metrics introduced as stated in the Public Interfaces section.
- kafka.server.ReplicaManager.PendingReassignmentCount: this metric records the count of partitions that are currently being reassigned onto this broker.
- kafka.server.BrokerTopicMetrics.ReassignmentBytesInPerSec: the inbound rate of reassignment related traffic in bytes/seconds aggregated per broker.
- kafka.server.BrokerTopicMetrics.ReassignmentBytesOutPerSec: the outbound rate of reassignment related traffic in bytes/seconds aggregated per broker.
Compatibility, Deprecation, and Migration Plan
As mentioned before we change the behavior of *.replication.throttled.rate and *.replication.throttled.replicas too. These shouldn't have a big effect generally as *.replication.throttled.rate was mostly used for reassignment throttling anyway.
Rejected Alternatives
I considered leaving the current behavior of the current configurations as is but in reality people don't really want to throttle any kind of non-ISR traffic. There are some tools that are building upon the existing configs to provide this experience but from their perspective it is likely to be a small change to fully utilize this new behavior.