THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
| Table of Contents |
|---|
Status
Current state: "Under Discussion"
Discussion thread: here
Voting thread: here
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation
Many times, Kafka brokers in production crash with "Too many open files" error or "Out of memory" errors because some Kafka topics have a lot of segment files as a result of small segment.ms or segment.bytes. These two configuration can be set by any user who is authorized to create topic or modify topic configuration.
To prevent these two configuration from causing Kafka broker crash, they there should have be a minimum value that is big enoughfor these configuration.
Public Interfaces
This KIP proposes to add these two broker configuration:
min.topic.segment.msmin.topic.segment.bytes
And also, this KIP proposes changes to the validation logic of KIP affects the following topic configuration:
segment.mssegment.bytes
Proposed Changes
This KIP proposes to add the following minimum values to these topic dynamic broker configuration:
| Configuration | Type | Current minimum Default value | Proposed minimum value | Documentation |
|---|---|---|---|---|
| min.topic.segment.ms0 | long | 3600000 | The minimum value for topic configuration segment.ms. | |
| min.topic.segment.bytes0 | int | 1048576 | The |
...
| minimum value for topic configuration segment.bytes. |
When provided via CreateTopics or AlterConfigs API, the validation of the following topic configuration changes as follow:
| Topic configuration | Validation |
|---|---|
segment.ms | If the value of segment.ms is less than the value of min.topic.segment.ms, Kafka broker will reject it by responding with InvalidConfigurationException |
segment.bytes | If the value of segment.bytes is less than the value of min.topic.segment.bytes, |
...
| Kafka broker will reject it by |
...
responding with InvalidConfigurationException |
Compatibility, Deprecation, and Migration Plan
Kafka brokers that have existing topics with segment.ms and/or segment.bytes smaller than the proposed minimum would have a warning in the log.
Rejected Alternatives
- If a topic has
segment.msthat is less thanmin.topic.segment.ms, Kafka broker will log a warning on startup "Topic X has segment.ms (Y) that is less than min.topic.segment.ms (Z)." - If a topic has
segment.bytesthat is less thanmin.topic.segment.bytes, Kafka broker will log a warning on startup "Topic X has segment.bytes (Y) that is less than min.topic.segment.bytes (Z)." - This way, the proposed changes will not cause issue on the existing topic, but still raise attention to topics with small segment sizes.
Rejected Alternatives
- Set a fixed Add new broker configuration to specify the allowed minimum value for segment.ms and segment.bytes – This will add another 2 configuration that will not be used much. And most people want Kafka to provide a safe default anyway.
...
- .bytes. The reason this approach is rejected is because there are use cases that requires setting segment.ms or segment bytes to very small value. For example: deleting all records in a topic by setting segment.bytes to 0 temporarily.