...

This KIP focuses on the changes required for use by the Kafka Controller's __cluster_metadata topic partition but we should be able to extend it to use it in other internal topic partitions like __consumer_offsets and __transaction.

Terminology

1. Kafka Controller

...

1. : The component that generates snapshots, reads snapshots and reads logs for voter replicas of the topic partition __cluster_metadata.
2. Active Controller

...

1. : The Kafka Controller that is also a leader of the __cluster_metadata topic partition. This component is allowed to append to the log.
2. Metadata Cache

...

1. : The component that generates snapshots, reads snapshots and reads logs for observer replicas of the topic partition __cluster_metadata. This is needed to reply to Metadata RPCs, for connection information to all of the brokers, etc.

Proposed Changes

The __cluster_metadata topic will have snapshot as the cleanup.policy. The Kafka Controller will have an in-memory representation of the replicated log up to at most the high-watermark. When performing a snapshot the Kafka Controller will serialize this in-memory state to disk. This snapshot file on disk is described by the largest offset and epoch from the replicated log that has been included.

...