Page History

...

If the Kafka Controller generates a snapshot too frequently then it can negatively affect the performance of the disk. If it doesn't generate a snapshot often enough then it can increase the amount of time it takes to load its state into memory and it can increase the amount space taken up by the replicated log. The Kafka Controller will have a new configuration option controller.snapshot.minimummin.cleanable.recordsratio.  If the number of snapshot records that have changed (deleted or modified) between the latest snapshot and the highcurrent in-watermark memory state is greater than controller.snapshot.min.minimumcleanable.recordsratio, then the Kafka Controller will perform a new snapshot.

Note that new snapshot records don't count against this ratio. If a new snapshot record was added since that last snapshot then it doesn't affect the dirty ratio. If a snapshot record was added and then modify or deleted then it counts against the dirty ratio.

When to Increase the Log Begin Offset

...

1. The __cluster_metadata topic will have a cleanup.policy value of snapshot. This configuration can only be read. Updates to this configuration will not be allowed.

2. controller.snapshot.minimum.records is the minimum number of records committed after the latest snapshot before the Kafka Controller and Metadata Cache will perform another min.cleanable.ratio - The minimum ratio of snapshot records that have to change before generating a snapshot. See section "When to Snapshot". The default is 16384 records.5 (50%).

3. maxcontroller.replicationlbo.lag.time.max.ms - The maximum amount of time that leader will wait for an offset to get replicated to all of the live replicas before advancing the LBO. See section “When to Increase the Log Begin Offset”. The default is 7 days.

...