Page History

...

For log compaction enabled topic, Kafka today uses “min.cleanable.dirty.ratio” and "min.compaction.lag.ms" to determine what log segments it need needs to pick up for compaction. "min.compaction.lag.ms" marks a log segment uncleanable until the segment is sealed and remains un-compacted for the specified "lag". The detailed information can be found in KIP-58.   “min.cleanable.dirty.ratio” is used to determine the eligibility of the entire partition for log compaction. Only log partitions whose dirty ratio is higher than “min.cleanable.dirty.ratio” are picked up by log cleaner for compaction.  In addition, when log cleaner performs compaction on a log partition, there is no guarantee it will compact all cleanable segments determined by "min.compaction.lag.ms". On each compaction run, log cleaner will build an offsetmap, the number of records that can be fit into the inserted to offsetmap also limit the number of log segments that can be compacted. In summary, with these two compaction configurations,  Kafka cannot enforce a timely log compaction.

...