...

Observer controllers are controllers who are either not part of the static voter set in kraft.version=0, or controllers who are not part of the bootstrap voter set defined by --initial-controllers or --standalone in kraft.version=1 (i.e. controllers who format today with --no-initial-controllers). Non-observer/"bootstrap controllers" are therefore either part of the static voter set in kraft.version=0, or controllers who format with --initial-controllers or --standalone  in a dynamic quorum setup. This distinction is important because observer controllers are not responsible for the initial KRaft leader election, but bootstrap controllers are.

We can still enforce that bootstrap controllers must have formatted (and therefore persisted a cluster id to meta.properties) prior to starting kafka. The validation of meta.properties during startup described above can be done whenever the node is a bootstrap controller (i.e. part of the static voters config, or if a 0-0.checkpoint exists with a VotersRecord). We still need to do this validation mainly for kraft.version=0 clusters with newer software versions but an older MV (kraft.version=1 clusters require formatting of at least one node to elect a leader). Otherwise, these clusters will have no way to persist a cluster id to meta.properties if the operator skips formatting on all nodes (which would be possible without this requirement).

Proposed Changes

Remove the requirement of brokers and observer controllers to format before starting kafka

• After KIP-1286, kafka operators no longer need to format all nodes. They only need to format the "bootstrap" controller nodes before starting kafka.
  • "Bootstrap controllers" are the nodes listed the controller.quorum.voters  static config when using a static quorum, or the controllers who format with --initial-controllers/--standalone  in a dynamic quorum setup.
  • Formatting brokers and observer controllers is now optional. Failing to run kafka-storage format  on these nodes before starting kafka will no longer crash startup.

meta.properties will be written during kafka broker/controller startup if it doesn't exist already (from formatting)

...