Current state: "Under Discussion"
Discussion thread: here
Voting thread: here
JIRA: KAFKA-7760
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Many times, Kafka brokers in production crash with "Too many open files" error or "Out of memory" errors because some Kafka topics have a lot of segment files as a result of small segment.ms
or segment.bytes
. These two configuration can be set by any user who is authorized to create topic or modify topic configuration.
To prevent these two configuration from causing Kafka broker crash, there should be a minimum value for these configuration.
This KIP proposes to add these two broker configuration:
min.topic.segment.ms
min.topic.segment.bytes
And also, this KIP proposes changes to the validation logic of the following topic configuration:
segment.ms
segment.bytes
This KIP proposes to add the following dynamic broker configuration:
Configuration | Type | Default value | Documentation |
---|---|---|---|
min.topic.segment.ms | long | 3600000 | The minimum value for topic configuration segment.ms. |
min.topic.segment.bytes | int | 1048576 | The minimum value for topic configuration segment.bytes. |
When provided via CreateTopics or AlterConfigs API, the validation of the following topic configuration changes as follow:
Topic configuration | Validation |
---|---|
segment.ms | If the value of segment.ms is less than the value of min.topic.segment.ms , Kafka broker will reject it by responding with InvalidConfigurationException |
segment.bytes | If the value of segment.bytes is less than the value of min.topic.segment.bytes , Kafka broker will reject it by responding with InvalidConfigurationException |
segment.ms
that is less than min.topic.segment.ms
, Kafka broker will log a warning on startup "Topic X has segment.ms (Y) that is less than min.topic.segment.ms (Z)." segment.bytes
that is less than min.topic.segment.bytes
, Kafka broker will log a warning on startup "Topic X has segment.bytes (Y) that is less than min.topic.segment.bytes (Z)."