...

With KRaft, Kafka added a new controller quorum to the cluster. This quorum These controllers needs to be available able to commit records for Kafka to be available. We are going One way to measure availability, is by periodically causing the high-watermark and the last committed offset to increase. Monitoring service services can compare that these last committed offset against every controller and brokeroffsets are advancing. They can also use these metrics to check that all of the brokers and controllers are relatively within each other's offset.

Public Interfaces

Cluster Metadata Records

...

{
  “apiKey”: TBD,
  “type”:  “metadata”,
  “name”: “NoopRecord”,
  “validVersions”: “0”,
  “flexibleVersions”: “0+”,
  “fields”: []
}

...

1. confluent.metadata.monitor.write.ms - Frequency for writing NoopRecord to the cluster metadata log.

Proposed Changes

Active Controller

The active controller will increase the LEO and high-watermark by periodically writing a noop record (NoopRecord) to the metadata log. The active controller will write this new record only if the IBP and metadata.version supports this feature. See the backward compatibility section for details.

The implementation will only append the NoopRecord, if the LEO wasn’t already advanced in the defined period.

Controllers and Brokers

In both the controller and broker, the metadata replaying code will be extended to ignore NoopRecordDescribe the new thing you want to do in appropriate detail. This may be fairly extensive and have large subsections of its own. Or it may be a few sentences. Use judgement based on the scope of the change.

Compatibility, Deprecation, and Migration Plan

...