You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 5 Next »

This page is meant as a template for writing a KIP. To create a KIP choose Tools->Copy on this page and modify with your content and replace the heading with the next KIP number and a description of your issue. Replace anything in italics with your own description.

Status

Current state: Under Discussion

Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]

JIRA: here 

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

KIP-226 added support for dynamic update of  broker configuration.  In this KIP, we propose to extend the support to dynamic update of a group of operationally important broker configs which will benefit the replication process.

Proposed Changes

ConfigUse case
fetch.max.bytesMaximum bytes expected for the entire fetch response.
failed.authentication.delayConnection close delay on failed authentication.
replica.fetch.response.max.bytesMaximum bytes expected for the entire fetch response.
replica.fetch.wait.max.msMax wait time for each fetcher request issued by follower replicas.
follower.replication.throttled (new config to add)Should all replication follower throttle
leader.replication.throttled (new config to add)Should all replication leader throttle


  • fetch.max.bytes: ReplicaManager will multi-inherit from BrokerReconfigurable but no reconfigurable interfaces need to be implemented since the fetcher request will calculate fetchMaxBytes as the minimum of fetchRequest.maxBytes and config.fetchMaxBytes
  • failed.authentication.delay: SocketServer already implemented the Reconfigurable interface.  Setter method will be added in both Processor and Selector and update the Selector::failedAuthenticationDelayMsProp variable in the Reconfigure callback.
  • replica.fetch.response.max.bytes: The two class ReplicaAlterLogDirsThread and ReplicaFetcherThread read this config as their class variable when they initialize. So they will multi-inherit from BrokerReconfigurable and implemented setter method to modify the class variable in the Reconfigure callback.

Public Interfaces

This KIP introduces two new broker level configs related to replication throttle:

  1. follower.replication.throttled - Should all replication follower throttle
  2. leader.replication.throttled - Should all replication leader throttle

Currently, we have two similar topic level configs:

leader.replication.throttled.replicas: A list of replicas for which log replication should be throttled on the leader side. The list should describe a set of replicas in the form [PartitionId]:[BrokerId],[PartitionId]:[BrokerId]:... or alternatively the wildcard '*' can be used to throttle all replicas for this topic.

follower.replication.throttled.replicas: A list of replicas for which log replication should be throttled on the follower side. The list should describe a set of replicas in the form [PartitionId]:[BrokerId],[PartitionId]:[BrokerId]:... or alternatively the wildcard '*' can be used to throttle all replicas for this topic.

However, sometimes operational people may want to throttle all the replication leader / follower. It's hard for them to achieve this using the topic level replication throttles.


Compatibility, Deprecation, and Migration Plan

  • There won't be any impact on existing users.
  • There won't be any change of current behavior.
  • No migration tool required

Rejected Alternatives

  • No labels