...

• Extra configs: two extra config will be added, called leader.reassignment.throttled.rate and follower.reassignment.throttled.rate. Please see proposed changes for more info.
• Disabled Existing configs: leader.replication.throttled.replicas and follower.replication.throttled.replicas would be disabled after this change.
• Metrics:
  • Count of partitions being reassigned per broker
  • Read and write rate for partition reassignment per broker in bytes/second
• keep their existing behavior and wouldn't be deprecated.
• Reassignment tool: the tool would use the new configs going forward instead of the old ones.

Proposed Changes

Behavior

...

of Existing Configs

The primary targets areWe would keep leader.replication.throttled.replicas and follower.replication.throttled.replicas. These will be rendered obsolete as the replicas to throttle would be computed from the new LeaderAndIsr targetReplicas field that KIP-455 introduces. The reasoning for computing these automatically is makes the administrator's life easier by eliminating some manual steps and the possibility of misconfiguration.  Also by disabling this config we won't allow throttling non-reassignment traffic because ISR replicas should have the bandwidth to stay in-sync and non-ISR replicas should also have the bandwidth to catch up. Therefore these configs won't have any effect after this KIP.Secondly the behavior of leader.replication.throttled.rate and follower.replication.throttled.rate will change slightly as a consequence of the above change. They will still mean leader and follower throttle rate but in this case will only be applied to reassignment and not to replication in general. because there are still scenarios where generic replication throttling is needed. One such case is a bootstrapping broker where a lot of follower partitions need to catch up with their leader but they're using up the bandwidth from the leader partitions on that broker. In this case it is still useful to set follower replication throttling so the leaders can stay in-sync. (Furthermore as a possible improvement to bootstrapping it might be useful to allow only a subset of follower partitions to replicate at a time but this falls out of the scope of this KIP.)

New Configs

We will introduce two new configs in order to eventually replace *.replication.throttled.rate. The reason for this is that their name don't express their functionality anymore and therefore should be replaced. From the user's perspective configuring *.replication.throttled.rate or the new configs would be equivalent but in the first case they would got a warning logged.

The new configs are:

...

Reassignment Tool

The --throttle option's behavior in kafka-reassign-partitions.sh would change as it would use the new configs going forward. If there is need for reproducing the old behavior then it would still be possible by calling kafka-configs.sh manually before and after the reassignment to set the correct replication throttling.

Throttling Calculation

The quota calculation method that is introduced in KIP-73 wouldn't change in principal but we will only apply it to the calculated (reassigned) replicas accordingly. This has the benefit that we don't need to change the recommendations in KIP-73.

Updating

...

Reassignment Throttling

At this point this KIP doesn't aim to add new AdminClient RPC call as the config value can be changed by the IncrementalAlterConfigs API.

Compatibility, Deprecation, and Migration Plan

As mentioned before we change the behavior of *.replication.throttled.rate and *.replication.throttled.replicas too. These shouldn't have a big effect generally as *.replication.throttled.rate was mostly used for reassignment throttling anyway.

Rejected Alternatives

I considered leaving the current behavior of the current configurations as is but in reality people don't really want to throttle any kind of non-ISR traffic. There are some tools that are building upon the existing configs to provide this experience but from their perspective it is likely to be a small change to fully utilize this new behaviorThe only change which needs to be mentioned is the tooling change. With this we'll change the --throttle option's behavior. If for some reason the old behavior is needed it can be reproduced by calling kafka-configs.sh manually before and after the reassignment with the intended parameters.