Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents

This page is meant as a template for writing a KIP. To create a KIP choose Tools->Copy on this page and modify with your content and replace the heading with the next KIP number and a description of your issue. Replace anything in italics with your own description.


Status

Current state:   [One of "Under Discussion", "Accepted", "Rejected"]

Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]  

JIRA: here [Change the link from KAFKA-1 to your own ticket]
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).  Kafka-14768


Motivation

Sometimes, application's threads will block for max.block.ms to send records using KafkaProducer#send. It exhausted threads of whole system for the time in some cases.

When application try to reduce the max.block.ms to decrease the blocking time. Thus, they will find they couldn't change the value to any one which is smaller than the time costed for metadata's fetch. What's more, metadata's fetch is one heavy operation which cost a lot of time.

...

if we need N (2<N<5) seconds for metadata's fetch: 

Cases to send record.

max.block.ms

includeWaitTimeOnMetadataInMaxBlockTime


maxWaitTimeMsOnMetadata

case 1 success

10 seconds

default value: false (no set)

default value: 60 seconds (no set)

case 2 fail to send

1 seconds

default value: false (no set)

default value: 60 seconds (no set)

case 3 success 

10 seconds

true

default value: 60 seconds (no set)

case 4 success

1 seconds

true

5 seconds

case 5 fail to send

1 seconds

true

1 seconds


Rejected Alternatives

One alternative is that providing new method to complete the metadata fetch not controlled by max.block.ms and user should call it before sending any record. For example, user can call it before marking the service ready.

...