Page History

...

• Rough design:
  • Introduce a new Metadata Version that supports a ClusterIDRecord.
  • Brokers/observers can start KRaft with no cluster id, and rely on metadata publishing pipeline to discover it in-memory
    • Upon discovering the cluster ID for the first time, these nodes need to persist this to meta.properties, and update the raft client in-memory.
    • Ideally, starting KRaft with no cluster id should only be allowed the "first time" (i.e. if the cluster metadata partition doesn't exist on the node yet).
  • Upon restart, nodes need to compare the cluster id from the metadata partition with their local meta.properties . The value in the metadata partition takes precedence.
    • If these values are different, log an error or crash.
    • If cluster ID exists in the metadata partition but not in meta.properties , write it to meta.properties .
  • Bootstrap controllers can add a mandatory “cluster id” record during formatting, or the initial leader can randomly generate a UUID as part of bootstrap metadata records write
  • Upon restart, in terms of ensuring a node does not talk to another cluster during its lifetime in the presence of deletions/changes to meta.properties , this approach is no better than Option 1 unless the local node can use the cluster id value in its local metadata partition BEFORE contacting the leader and learning of the HWM. This means that the local a node needs to read the its local metadata log BEFORE discovering the HWM from the leader during startup to retrieve the cluster id, which may be a performance concern.
    • The local node does not know if the cluster id record in its metadata log is committed or not until it contacts the leader and learns the HWM. However, in order to ensure the local node does not talk to another cluster, it needs to provide a cluster ID in its fetch request to discover the HWM. This is a circularity.
      • However, in practice this should be okay. If ClusterIdRecord with value X was written as part of the bootstrap metadata records write (this is the only way it is possible for a node to have this record in its metadata partition), this means that a KRaft quorum with cluster id X was formed. The offset might be uncommitted and change if the bootstrap write fails or leadership changes, but consensus for the value X as the cluster ID has already been achieved upon electing a leader.
• Pros:
  • Fetch replication automatically handles persistence of the cluster id for each local node
  • Raft module remains independent from metadata module in that KRaft is only responsible for consensus. ClusterID is simply another piece of metadata on which Kraft achieves consensus
• Cons:
  • Currently, KRaft client also needs to be aware of the cluster ID for its own RPC handling, but the raft module does not decode metadata records
    • Can duplicate the cluster ID as a control record
    • Having a mechanism for “pushing-down” cluster ID from metadata to raft may be complicated.
    • We can duplicate data and have a raft level control record for cluster ID.
    • The fact that the raft client does currently do validation on cluster id does make it unique (i.e. it is used by both metadata and raft layers to prevent nodes from talking to different “clusters”, which is an argument for option 1).
      • For example, if Kraft was used to replicate other data besides the metadata partition, there would still be a concept of cluster id, which needs to be the same across all partitions on the node being managed by Kraft.

...