THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
This page is meant as a template for writing a KIP. To create a KIP choose Tools->Copy on this page and modify with your content and replace the heading with the next KIP number and a description of your issue. Replace anything in italics with your own description.
Status
Current state: Under Discussion
Discussion thread: here
JIRA: here
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation
KIP-226 added support for dynamic update of broker configuration. In this KIP, we propose to extend the support to dynamic update of a group of operationally important broker configs which will benefit the replication process.
Proposed Changes
| Config | Use case |
|---|---|
| fetch.max.bytes | Maximum bytes expected for the entire fetch response. |
| failed.authentication.delay | Connection close delay on failed authentication. |
| replica.fetch.response.max.bytes | Maximum bytes expected for the entire fetch response. |
| replica.fetch.wait.max.ms | Max wait time for each fetcher request issued by follower replicas. |
| follower.replication.throttled (new config to add) | Should all replication follower throttle |
| leader.replication.throttled (new config to add) | Should all replication leader throttle |
fetch.max.bytes: ReplicaManager will multi-inherit from BrokerReconfigurable but no reconfigurable interfaces need to be implemented since the fetcher request will calculate fetchMaxBytes as the minimum of fetchRequest.maxBytes and config.fetchMaxBytes
failed.authentication.delay: SocketServer already implemented the Reconfigurable interface. Setter method will be added in both Processor and Selector and update the Selector::failedAuthenticationDelayMsProp variable in the Reconfigure callback.
replica.fetch.response.max.bytes: The two class ReplicaAlterLogDirsThread and ReplicaFetcherThread read this config as their class variable when they initialize. So they will multi-inherit from BrokerReconfigurable and implemented setter method to modify the class variable in the Reconfigure callback.
replica.fetch.wait.max.ms: The changes are similar to replica.fetch.response.max.bytes
follower.replication.throttled: A new class variable LeaderReplicationThrottledProp will be added to ReplicationQuotaManager. Similar to leader.replication.throttled.rate, the BrokerConfigHandler will call updateBrokerThrottle() and update this class variable upon receiving the config change notification from ZooKeeper. ReplicationQuotaManager::isThrottled() will have its logic modified and judge based on if all the replication follower is throttled.
leader.replication.throttled: Similar to follower.replication.throttled
Public Interfaces
This KIP introduces two new broker level configs related to replication throttle:
follower.replication.throttled - Should all replication follower throttle
leader.replication.throttled - Should all replication leader throttle
Currently, we have two similar topic level configs:
leader.replication.throttled.replicas: A list of replicas for which log replication should be throttled on the leader side. The list should describe a set of replicas in the form [PartitionId]:[BrokerId],[PartitionId]:[BrokerId]:... or alternatively the wildcard '*' can be used to throttle all replicas for this topic.
follower.replication.throttled.replicas: A list of replicas for which log replication should be throttled on the follower side. The list should describe a set of replicas in the form [PartitionId]:[BrokerId],[PartitionId]:[BrokerId]:... or alternatively the wildcard '*' can be used to throttle all replicas for this topic.
However, sometimes operational people may want to throttle all the replication leader / follower. It's hard for them to achieve this using the topic level replication throttles.
Compatibility, Deprecation, and Migration Plan
- There won't be any impact on existing users.
- There won't be any change of current behavior.
- No migration tool required
To discuss
- What should be the type / format of follower.replication.throttled and leader.replication.throttled.replicas, a boolean or a list such as "[partitionId]:[brokerId], [partitionId]:[brokerId]"?
- For the two new broker level configurations follower.replication.throttled and leader.replication.throttled.replicas, we should think about their precedence compared to the existing topic level configs leader.replication.throttled.replicas and leader.replication.throttled.replicas. When the broker level throttle and topic level throttle both are both set, which will dominate?