...

Observer controllers are controllers who are either not part of the static voter set in kraft.version=0, or controllers who are not part of the bootstrap voter set defined by --initial-controllers or --standalone in kraft.version=1 (i.e. controllers who format today with --no-initial-controllers). Non-observer/bootstrap controllers are therefore either part of the static voter set in kraft.version=0, or controllers who format with --initial-controllers or --standalone. This distinction is important because observer controllers are not responsible for the initial KRaft leader election, but bootstrap controllers are.

After this KIP, brokers and observer controllers are no longer guaranteed to have persisted a meta.properties with a cluster.id  value prior to starting kafka if the intended metadata version of the cluster supports this feature. Even after this KIP, we We can still enforce that bootstrap controllers must have formatted (and therefore persisted a cluster id to meta.properties) prior to starting kafka. The validation of meta.properties during startup described above can be done whenever the node is a bootstrap controller (i.e. part of the static voters config, or if a 0-0.checkpoint exists with a VotersRecord). We still need to do this validation mainly for kraft.version=0 clusters with newer software versions but an older MV (kraft.version=1 clusters require formatting of at least one node to elect a leader). Otherwise, these clusters will have no way to persist a cluster id to meta.properties if the operator skips formatting on all nodes (which would be possible without this requirement).

Proposed Changes

meta.properties can be written during kafka broker/controller startup if it doesn't exist already (from formatting)

• During startup of the KafkaRaftServer, we attempt to read the meta.properties file
• If meta.properties  does not exist, write it with meta.properties V2 with node.id  and directory.id
  • This means either this node skipped formatting, or the file/disk was lost
• If meta.properties exists without a cluster.id and is V2, it will be discovered later (described below)
  • One correctness invariant of this feature is that updating the cluster.id in-memory and persisting it to meta.properties  must be atomic.
• If meta.properties exists with a cluster.id, kafka behaves as it does today:
  • The node assumes it to be correct and passes it to KRaft
  • If this ID doesn't match the KRaft leader's the leader will reject these requests

...