THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
Versions Compared
compared with
Key
- This line was added.
- This line was removed.
- Formatting was changed.
...
So, this KIP try to reach the goal we can change the max.block.ms to wanted smaller value without worry about the metadata's fetch.
...
Add two configures with tiny code changes related which control the timeout in KafkaProducer#send
1. Two configures added
producerProducer's configure. | configure item. | default value |
includeWaitTimeOnMetadataInMaxBlockTime | max.block.ms.include.metadata | false |
maxWaitTimeMsOnMetadata | max.block.metadata.ms | <max.block.ms |
2. Code changes
By default, includeWaitTimeOnMetadataInMaxBlockTime is true, all of the behaviors are not changed.
...
Cases to send record. | max.block.ms | includeWaitTimeOnMetadataInMaxBlockTime(max.block.ms.include.metadata) | maxWaitTimeMsOnMetadata(max.block.metadata.ms) |
case 1 success | 10 seconds | default value: false (no set) | default value: 60 seconds (no set) |
case 2 fail to send | 1 seconds | default value: false (no set) | default value: 60 seconds (no set) |
case 3 success | 10 seconds | true | default value: 60 seconds (no set) |
case 4 success | 1 seconds | true | 5 seconds |
case 5 fail to send | 1 seconds | true | 1 seconds |
Rejected Alternatives
Alternative 1:
Provide One alternative is that providing new method to complete the metadata fetch not controlled by max.block.ms and user should call it before sending any record. For example, user can call it before marking the service ready.
The alternatives can solve the issue and also solve the first record's slow latency issue. but if user only have interesting to reduce the blocking time without care about it. It isn't the best solution for this requirement due to user may forget it to call it before any sending or don't aware to call another method to solve the issue.
Alternative 2:
Refer to Another possible alternative is https://cwiki.apache.org/confluence/display/KAFKA/KIP-286. The KIP's goal is that "We will change the behavior of producer.send() so that it does not block on metadata update".
I think the metadata's blocking is still neededmust-have one. the thing we can do is to move the blocking before producer.send(). Thus, the KIP don't solve the issue from this point.