Page History

...

The reason for this KIP is to remove the requirement of brokers nodes needing to run kafka-storage format  before starting Kafka. When running kafka-storage format , nodes are required to supply a --cluster-id  argument, which represents the cluster ID to which the node belongs, and this is persisted to the node's meta.properties  file. Below are the other data that each node persists to disk upon invoking kafka-storage format :

...

Remove the requirement of nodes to format before starting kafka

After KIP-1262, kafka operators no longer need to invoke kafka-storage format to start kafka. Clusters who skip formatting essentially have a bootstrapping "state" of the latest MetadataVersion and a ClusterIdRecord who is generated by the initial active controller.

Clusters who skip formatting will also have a static KRaft quorum, which is explained in the below section, so controller.quorum.voters must be defined. Operators can still end up with a dynamic quorum without formatting because static to dynamic quorum upgrades are supported.

Operators can still format clusters the same way as prior to this KIP. However, the --cluster-id  flag now becomes optional.

...

When --cluster-id

...

is specified,

...

write meta.properties

...

V2 with it. When --cluster-id  is not specified, and the formatter's resolved MV supports ClusterIdRecord, write meta.properties V2 with an empty cluster.id.

...

 If the formatter's resolved MV does not support ClusterIdRecord, formatting will fail.

When is formatting still required before starting kafka

There are a few situations where operators still need to format nodes before starting kafka, but this only applies to bootstrap controllers. Bootstrap controllers are those who participate in the initial KRaft leader election. These are the nodes specified by controller.quorum.voters in a static quorum, and the nodes who format with --standalone/--initial-controllers  in a dynamic quorum.

Formatting of bootstrap controllers is still required when operators want to:

• Specify bootstrap SCRAM credentials
• Specify bootstrap non-default feature levels or metadata version
• Bootstrap a dynamic quorum (kraft.version=1) cluster.

meta.properties will be written during kafka broker/controller startup if it doesn't exist already (from formatting)

During startup of the KafkaRaftServer, we attempt to read the meta.properties file:

• If meta.properties does not exist and the node is a broker/observer controller, write meta.properties V2 with node.id  and directory.id
  • This means either this node skipped formatting, or the file/disk was lost
• If meta.properties does not exist exists and the node is a bootstrap controller, throw a runtime exceptionThis is v1, do the same behavior validations as we do today in kafka today
• If meta.properties exists without a cluster.id and is V2, it will be discovered later (described below)
  • One correctness invariant of this feature is that updating the cluster.id in-memory must occur after persisting it to meta.properties.
• If meta.properties exists with a cluster.id, kafka behaves as it does today:
  • The node assumes it to be correct and passes it to KRaft
  • If this ID doesn't match the KRaft leader's ID, the leader will reject requests from the node

If a broker/observer controller has already written a cluster id to

...

its meta.properties, either from formatting or discovering it from the cluster metadata, it is impossible for it to learn of another cluster id via Fetch/FetchSnapshot.

...

For the broker (with clusterid = X) to receive a non-error FetchResponse with metadata records (which would be the only way to learn of a different ClusterIdRecord), the KRaft leader (clusterid = Y) must either receive a request without clusterid, or a request whose clusterid is Y. The broker fulfills neither of these conditions.

...

This case could happen when bootstrap endpoints point to the wrong cluster during restart of a node. KRaft's own clusterid checks would mean startup of this node would time out and shut down the node because it is not able to contact the quorum of another cluster.

...

Introduce a new MetadataVersion

...

and the ClusterIdRecord

One invariant of this feature is that if the persisted MV supports this feature, a ClusterIdRecord must also be persisted

...

. This is enforceable along the write-path for MV, which occurs at the following points: formatting a node (specifically controllers who can become leader), the bootstrap metadata write of the initial active controller, and upgrading the MV using kafka-features upgrade

...

.

• During formatting, nodes must resolve a MV with which to format. This comes from the --release-version/--feature flag and defaults to the latest production MV. If the MV at format time supports this feature, a ClusterIdRecord must be written as part of the bootstrap metadata checkpoint (0-0.checkpoint post KIP-1170)
• The first active controller will write the ClusterIdRecord + MV as part of the bootstrap metadata records write if the MV supports this feature
• During MV upgrades, successfully upgrading the MV to one that supports this feature requires successfully committing a ClusterIdRecord alongside the new MetadataVersion feature record.

There is a precedent already for this kind of invariant which is enforced along the write path with kraft.version and

...

the VotersRecord 

When nodes discover cluster.id from the metadata publishing pipeline, they persist it to meta.properties + update KRaft

Nodes whose meta.properties do not contain cluster.id  will discover this value via the metadata publishing pipeline

...

. This can be implemented as a MetadataPublisher that registers to the MetadataLoader.

• It is the responsibility of this MetadataPublisher to persist a discovered value of cluster.id to all meta.properties files before making the cluster.id available to in-memory data structures.
  • Because of JBOD, nodes can have many log.dirs that each have a meta.properties file (or we need to write each instance of meta.properties V2 during startup without cluster.id if we skipped formatting). 
• The MetadataPublisher is the single writer of cluster.id (unless it is already set by formatting, in which case there are no writers to cluster.id), both in persisting the value to meta.properties , and writing the in-memory representation of cluster.id
  • There are many readers of this value, but most of them can block their initialization until this value is discovered for the sake of simplicity.
  • The KafkaRaftClient and QuorumController are the readers who need to handle the concept of an empty/not-set cluster id. 

...

Remove the requirement for formatting for brokers and observer controllers, but still require it for bootstrap controllersBootstrap controllers are those who participate in the initial KRaft leader election. These are the nodes specified by controller.quorum.voters in a static quorum, and the nodes who format with --standalone/--initial- controllers  in a dynamic quorum. 

The main reason we can relax this to say no nodes require formatting is that KRaft can elect a leader when all nodes do not have cluster.id defined.