Status

Current state: "Under Discussion"

Discussion thread: here

JIRA: KAFKA-13040

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Many times, Kafka brokers in production crash with "Too many open files" error or "Out of memory" errors because some Kafka topics have a lot of segment files as a result of small segment.ms or segment.bytes. These two configuration can be set by any user who is authorized to create topic or modify topic configuration.

To prevent these two configuration from causing Kafka broker crash, they should have a minimum value that is big enough.

Public Interfaces

This KIP affects the following topic configuration:

• segment.ms
• segment.bytes

Proposed Changes

This KIP proposes to add the following minimum values to these topic configuration:

The proposed minimum value is determined to avoid having too small segment files.

If segment.ms and/or segment.bytes value smaller than the minimum value is provided in CreateTopics or AlterConfigs command, then Kafka broker will reject it by sending InvalidConfigurationException.

Compatibility, Deprecation, and Migration Plan

Kafka brokers that have existing topics with segment.ms and/or segment.bytes smaller than the proposed minimum would have a warning in the log.

Rejected Alternatives

• Add new broker configuration to specify the allowed minimum value for segment.ms and segment.bytes – This will add another 2 configuration that will not be used much. And most people want Kafka to provide a safe default anyway.