Page History

...

Currently, Kafka relies on the operator/caller of kafka-storage format on all intended nodes in a cluster to generate consensus on the actual value of clusterID during format time. KRaft guarantees that upon forming a given quorum X, X's members will all have the same cluster id (or no cluster idwhich could clusterId = null) contained in meta.properties. Additionally, when a KRaft node sends a request with a non-null cluster id, this request is only handled if the request's cluster ID matches the handler's cluster ID.

...

Remove the requirement of brokers and observer controllers to format before starting kafka

• After KIP-12861262, kafka operators no longer need to format all nodes. They only need to format the "bootstrap" controller nodes before starting kafka.
  • "Bootstrap controllers" are the nodes listed the controller.quorum.voters  static config when using a static quorum, or the controllers who format with --initial-controllers/--standalone  in a dynamic quorum setup.
  • Formatting brokers and observer controllers is now optional. Failing to run kafka-storage format  on these nodes before starting kafka will no longer crash startup.

...

• Introduce a new MetadataVersion for this feature alongside a new metadata record called ClusterIdRecord
• One invariant of this feature is that if the persisted MV supports this feature, a ClusterIdRecord must also be persisted
  • This is enforceable along the write-path for MV, which occurs at two points: formatting a node (specifically controllers who can become leader) and upgrading the MV using kafka-features upgrade 
    • During formatting, nodes must resolve a MV with which to format. This comes from the --release-version/--feature flag and defaults to the latest production MV. If the MV at format time supports this feature, a ClusterIdRecord must be written as part of the bootstrap metadata checkpoint.
      • The first active controller will write the ClusterIdRecord + MV as part of the bootstrap metadata records write if the MV supports this feature
    • During MV upgrades, successfully upgrading the MV to one that supports this feature requires successfully committing a ClusterIdRecord alongside the new MetadataVersion feature record.
  • There is a precedent already for this kind of invariant which is enforced along the write path with kraft.version and the VotersRecord 
  However, kafka should still be able to handle the case where a leader is elected who does not have clusterId in meta.properties, which can occur if a majority of voters do not have a clusterIdThis is because KRaft does not need cluster.id
  •  
  in order to elect a leader (i.e. the cluster id for all voters can be empty and elect a leader)In this case, the active controller will write a cluster id record during the bootstrap metadata write.

When nodes discover cluster.id from the metadata publishing pipeline, they persist it to meta.properties

...