Status

Current state: ["Under Discussion"

Discussion thread: here

JIRA: KAFKA-2406

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

This KIP is related to KIP-4, KAFKA-1367 and KAFKA-2406. In the future, all the Kafka clients and admin tools will not access zookeeper directly. Therefore KAFKA-1367 was introduced to propagate the ISR change to all the brokers to sync up topic metadata on brokers. The approach is to create a ZK path to notify controller update ISR and send UpdateMetadataRequest to brokers. The issue is during a broker bounce or failure, the amount of ISR change are huge and overwhelm the controller. This KIP is to introduce a new config to batch the ISR update and throttle the ISR propagation rate.

Public Interfaces

val IsrChangePropagateIntervalMsProp = "isr.change.report.interval.ms"

val IsrChangePropagateIntervalMsDoc = "Specify the interval to propagate ISR change to the entire cluster. The ISR propagation involves zookeeper" +

          " path creation. Propagating ISR change too frequently might cause performance issue."

Default value for the config will be set to 5 seconds.

Proposed Changes

Add a new config IsrPropagateIntervalMs to KafkaConfig. The broker will only propagate ISR change at most every IsrPropagateIntervalMs. The propagation will create a sequential ZK path in zookeeper /isr_change_notification with data of all the partition whose ISR has changed since last propagation.

Compatibility, Deprecation, and Migration Plan

• What impact (if any) will there be on existing users?
• If we are changing behavior how will we phase out the older behavior?
• If we need special migration tools, describe them here.
• When will we remove the existing behavior?

Rejected Alternatives

If there are alternative ways of accomplishing the same thing, what were they? The purpose of this section is to motivate why the design is the way it is and not some other way.