Status

Current state: Under Discussion

Discussion thread: TODO

JIRA:  Unable to render Jira issues macro, execution error.

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

For the cluster metadata topic partition, snapshots are used to compact the log and remove redundant records. Kafka supports generating snapshots based on the amount of bytes appended to the log since the last snapshot. This works well to amortize the cost of generating snapshots.

In addition to compaction, snapshot can also be used as backups to the cluster metadata partition. To better support this case we should allow Kafka to generate snapshot based on time.

Public Interfaces

Configuration

The property metadata.log.between.snapshot.interval.ms will be added to the Kafka server configuration. This is the maximum amount of time that Kafka will wait to generate a snapshot if there are records in the log that are not included in the latest snapshot.

Proposed Changes

Both the KRaft brokers and controllers will generate a snapshot if there is a record in the log that is newer than the latest snapshot and has an append time older than metadata.log.between.snapshot.interval.ms milliseconds ago.

Compatibility, Deprecation, and Migration Plan

There are no compatibility, deprecation or migration implications. This feature only affects the local replica and doesn't have any effect on remote replicas.

Test Plan

We will add both unit and integration tests for this feature.

Rejected Alternatives

No alternative design or implementation where considered.