Page History

...

New producer config option is added:

• enable.mute.partition:  When set to ‘true’, Producer will call ProducerInterceptor and Partitioner initialization ProducerMuteManager during construction

Add a ProducerMuteManager class that manages the partition metadata related to this mechanism

...

We propose to add a configuration driven circuit breaking mechanism that allows Kafka client to ‘mute’ partitions when certain condition is met. The mechanism adds callbacks in Sender class workflow that allows to filtering partitions based on certain policy.

In addition to the interface, we provide a default implementation that uses failure rate as the condition. The client can choose proper implementation that fits a special failure scenario.Several parameters together defines the behavior of the default circuit breaker, Client-side custom implementation of Partitioner and ProducerInterceptor

• Customize the implementation of ProducerInterceptor, and choose the strategy to mute partitions.

• Customize the implementation of Partitioner, and choose the strategy to filtering partitions.

• The breaker is enabled only after certain number (producer.circuit.breaker.trigger.message.count) of messages are sent

• The breaker is triggered when failure rate partition exceeds a threshold (producer.circuit.breaker.trigger.failure.perc)

• Muted partition will be monitored and reset upon successful writes after a period (producer.circuit.breaker.mute.retry.interval)

• When producer.circuit.breaker.enable.mute.inflight.full is set and max.in.flight.requests.per.connection is set to 1 (sequential message), muting under infight congestion is enabled

Muting partitions have impact when the topic contains keyed message as messages will be written to more than one partitions during period of recovery. We believe this can be an explicit trade-off the application makes between availability and message ordering.

...