...

Compatibility, Deprecation, and Migration Plan

This section depends on which approach is chosen for the proposed changesSince this feature is associated with a new metadata record and MetadataVersion. Broker bootstrapping with cluster ID must be supported on all MVs < X where X is the first MV that supports this feature. Because some MetadataVersion is resolved during each node's formatting, we can determine at format time if a cluster id is needed.

Test Plan

• Unit tests
• Integration tests
• System tests to verify cross-software-version compatibility

Rejected Alternatives

Continue to persist cluster id in meta.properties but have KRaft discover it + persist it via FetchResponse

• Rough design:
  • Node can complete a future to allow this value to be discovered by readers outside of kraft layer who need it during startup
  • Raft layer is brought up early during startup, so it is fine to wait until this future completes to proceed with initializing the server
  • Brokers/observers can start Kafka with no cluster id, and rely on the fetch/another RPC response to discover it in-memory
    • If discovered, node persists cluster id to meta.properties during the startup process before in-memory readers of cluster id
• Pros:
  • Backwards compatibility is straightforward, since new nodes on old clusters keep using meta.properties for persisting cluster id
  • This functionality is not tied to a MetadataVersion, meaning that any Kafka broker/observer with a software version that supports this KIP can use it, rather than the whole cluster needing to be on some MV >= X.
  • Kraft can easily do its own cluster ID validation for its RPCs, since nodes receive cluster ID via the fetch response if they do not know it and can update that state in-memory + persist it
• Cons:
  • Since each local node's meta.properties  is its source of truth, if this file is deleted or is changed, it means the broker/observer controller can join another cluster.
    • This is no worse than what exists currently.