Page History

...

• Adding topic level configuration "max.compaction.lag.ms",  and corresponding broker configuration "log.cleaner.max.compaction.lag.ms", which is set to MAX_LONG by default. Kafka validates "max.compaction.lag.ms" is no less than "min.compaction.lag.ms".  A record may remain un-compacted for this max lag, after which the corresponding log partition becomes eligible for log compaction. This configuration only applies to topics that have compaction enabled. 

• Add the following metric:  

  1) kafka.log:type=LogCleaner,name=max-compaction-delay-secs
  type: gauge
  value: Math.max(now - earliest_timestamp_of_uncompacted_segment - max.compaction.lag.ms, 0)/1000
  This metric tells the max compaction delay after the time when a log is required for compaction as determined based on max.compaction.lag.ms and the time when the compaction is done for the log. 

...

• If a log partition already gets compacted once per day before this KIP,  setting the log compaction time interval to more than one day should have little impact on the amount of resource spent on compaction since the existing log compaction configuration (e.g., min dirty ratio) will trigger log compaction before "max.compaction.lag.ms".   The added metric "max-compaction-delay-secs" can be used to determine whether there are some partitions are actually determined by "max.compaction.lag.ms" to be compacted.  

...