Current state: Under Discussion
Discussion thread: here
JIRA: Kafka-14768
Sometimes, application's threads will block for max.block.ms to send records using KafkaProducer#send. It exhausted threads of whole system for the time in some cases.
...
So, this KIP try to reach the goal we can change the max.block.ms to wanted smaller value without worry about the metadata's fetch.
No public interface changed. Just change the inner implement of private method:
...
Add two new configure items for producer.
The changes can refer to the example PR: https://github.com/apache/kafka/pull/13335/files
...
When user set includeWaitTimeOnMetadataInMaxBlockTime to false, KafkaProducer#send will block maxWaitTimeMsOnMetadata for metadata's fetch and block max.block.ms for remaining operations.
If user want to use the feature, user can upgrade the client with the new configures set.
...
- What impact (if any) will there be on existing users?
no impact on existed users. - If we are changing behavior how will we phase out the older behavior?
no changing older behavior. - If we need special migration tools, describe them here.
no. - When will we remove the existing behavior?
no need to remove.
We can test with test matrix:
...
Cases to send record. | max.block.ms | includeWaitTimeOnMetadataInMaxBlockTime(max.block.ms.include.metadata) | maxWaitTimeMsOnMetadata(max.block.metadata.ms) |
case 1 success | 10 seconds | default value: false (no set) | default value: 60 seconds (no set) |
case 2 fail to send | 1 seconds | default value: false (no set) | default value: 60 seconds (no set) |
case 3 success | 10 seconds | true | default value: 60 seconds (no set) |
case 4 success | 1 seconds | true | 5 seconds |
case 5 fail to send | 1 seconds | true | 1 seconds |
One alternative is that providing new method to complete the metadata fetch not controlled by max.block.ms and user should call it before sending any record. For example, user can call it before marking the service ready.
...