Status

Current state: Under Discussion

Discussion thread: here  

JIRA:  Kafka-14768

Motivation

Sometimes, we found the users' functional interaction take a lot of time. At last, we figure out the root cause is that after we complete deploy or restart the servers.

The first message's delivery on each application server by kafka client will take much time.

After analyzing the source code about the first time's sending logic. The time cost is caused by the getting metadata before the sending. The latter's sending won't take the much time due to the cached metadata. The metadata's fetching logic is right and necessary. Thus, we still want to improve the experience for the first message's send/user first interaction.

So, This KIP try to raise one solution to improve it.

Public Interfaces

No public interface changed. Just change the inner implement of private method:

org.apache.kafka.clients.producer.KafkaProducer#doSend

Add two new configure items for producer.

Proposed Changes

The changes can refer to the example PR:   https://github.com/apache/kafka/pull/13320/files

Add two configures with tiny code changes related which control the timeout in KafkaProducer#send

1. Two configures added

2. Code changes

By default, includeWaitTimeOnMetadataInMaxBlockTime is true, all of the behaviors are not changed.

When user set includeWaitTimeOnMetadataInMaxBlockTime to false, KafkaProducer#send will block maxWaitTimeMsOnMetadata for metadata's fetch and block max.block.ms for remaining operations.

Compatibility, Deprecation, and Migration Plan

If user want to use the feature, user can upgrade the client with the new configures set.

If user don't have requirement for it, there isn't any need to do any change. What's more, new client version's upgrade also won't influence existed behavior.

• What impact (if any) will there be on existing users?  
  no impact on existed users.
• If we are changing behavior how will we phase out the older behavior?
  no changing older behavior.
• If we need special migration tools, describe them here.
  no.
• When will we remove the existing behavior?
  no need to remove.

Test Plan

We can test with test matrix:

if we need N (2<N<5) seconds for metadata's fetch, we will send record to test producer with different configures.

Case 2 and case 5 will fail to send records. All of others are success.

Rejected Alternatives

Alternative 1: