...

• Topic usage (number of topics, topic volume) is unpredictable.
• Availability of the latest content for all topics is more important than specific time- and size-based limits per topic.
• The disks should never ever run out of free space.
• Periodically deleting the globally oldest segments is a reasonable strategy to prevent the disks from running out of free space.

Public Interfaces

We propose adding a new broker configuration option:

val LogRetentionDiskUsagePercentProp = "log.retention.disk.usage.percent"
val LogRetentionDiskUsagePercentDoc = "The maximum percentage of disk space allowed to be in use (per-disk). Deletes oldest segments (across all topics) to maintain this usage ceiling."

...