...

Note that this protocol assumes something like the Kafka v2 log message format. It is not compatible with older formats because records do not include the leader epoch information that is needed for log reconciliation. We have intentionally avoided any assumption about the representation of the log and its semantics. This makes it usable both for internal metadata replication and (eventually) partition data replication.

A note on scope: This proposal is only concerned with the semantics and the management of the Metadata Quorum identified in the architecture in KIP-500. This includes the following:

• Specification of the replication protocol and its semantics
• Specification of the log structure and message schemas that will be used to maintain quorum state
• Tooling support to manage the quorum

This work does not define the messages that will be used by the Controller to maintain the state of the cluster. That will come in a follow-up KIP.

Key Concepts

If you are familiar with Kafka replication, most of the same concepts apply; however we replace ISR commit semantics with quorum commit semantics and add leader election.

...

1. The core requests have a field for the clusterId. This is validated when a request is received to ensure that misconfigured brokers will not cause any harm to the cluster.
2. All requests save Vote have a field for the leader epoch. Voters and leaders are always expected to ensure that the request epoch is consistent with its own and return an error if it is not.
3. We piggyback current leader and epoch information on all responses. This reduces latency to discover a leader change.

Note : This that this protocol is only concerned with leader election and log replication. It does not specify how new what log entries are appended to the leader's log nor how they will be received by the leader. Typically this would be through specific metadata management APIs. For example, KIP-497 adds an AlterISR API. When the leader of the metadata quorum (i.e. the controller) receives an AlterISR request, it will append an entry to its log.

...