...
Discussion thread: here
JIRA: KAFKA-13040 KAFKA-7760
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
...
To prevent these two configuration from causing Kafka broker crash, they there should have be a minimum value that is big enoughfor these configuration.
Public Interfaces
This KIP proposes to add these two broker configuration:
min.topic.segment.ms
min.topic.segment.bytes
This And also, this KIP proposes changes to the validation logic of the following topic configuration:
...
Compatibility, Deprecation, and Migration Plan
...
- If a topic has
segment.ms
that is less thanmin.topic.segment.ms
, Kafka broker will log a warning on startup "Topic X has segment.ms (Y) that is less than min.topic.segment.ms
...
- (Z)."
- If a topic has
segment.bytes
that is less thanmin.topic.segment.bytes
...
- , Kafka broker will log a warning on startup "Topic X has segment.bytes (Y) that is less than min.topic.segment.bytes (Z)."
- This way, the proposed changes will not cause issue on the existing topic, but still raise attention to topics with small segment sizes.
Rejected Alternatives
- Set a fixed minimum value for segment.ms and segment.bytes. The reason this approach is rejected is because there are use cases that requires setting segment.ms or segment bytes to very small value. For example: deleting all records in a topic by setting segment.bytes to 0 temporarily.
...