Status

Current state: ["Under Discussion"

Discussion thread: here

JIRA: KAFKA-2406

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

This KIP is related to KIP-4, KAFKA-1367 and KAFKA-2406. In the future, all the Kafka clients and admin tools will not access zookeeper directly. Therefore KAFKA-1367 was introduced to propagate the ISR change to all the brokers to sync up topic metadata on brokers. The approach is to create a ZK path to notify controller update ISR and send UpdateMetadataRequest to brokers. The issue is during a broker bounce or failure, the amount of ISR change are huge and overwhelm the controller. This KIP is to introduce a new config to batch the ISR update and throttle the ISR propagation rate.

Public Interfaces

val IsrChangePropagateIntervalMsProp = "isr.change.report.interval.ms"
val IsrChangePropagateIntervalMsDoc = "Specify the interval to propagate ISR change to the entire cluster." + 
                                      "The ISR propagation involves zookeeper path creation. Propagating ISR" +
                                      " change too frequently might cause performance issue."

 

Default value for the config will be set to 5 seconds.

Proposed Changes

Add a new config IsrPropagateIntervalMs to KafkaConfig. The broker will only propagate ISR change at most every IsrPropagateIntervalMs. The propagation will create a sequential ZK path in zookeeper /isr_change_notification with data of all the partition whose ISR has changed since last propagation.

Compatibility, Deprecation, and Migration Plan

This is a backward compatible change.

Rejected Alternatives

None

  • No labels