Status

Current state: Under DiscussionApproved

Discussion thread: here

JIRA: KAFKA-13883

...

Public Interfaces

Cluster Metadata Records

...

NoOpRecord

Add a new record to periodically advancement of the LEO and high-watermark. Controller or broker state will not change when applying this record. This record will not be included in the cluster metadata snapshot.

{
  “apiKey”: TBD,
  “type”: “metadata”,
  “name”: “NoopRecord”“NoOpRecord”,
  “validVersions”: “0”,
  “flexibleVersions”: “0+”,
  “fields”: []
}

...

1. kafka.controller:type=KafkaController,name=MetadataLastCommittedOffsetMetadataLastAppliedRecordOffset 
   Reports the offset of the last committed offset record consumed by the controller. For the active controller this may include uncommitted records. For the inactive controller this always includes committed records.
   
2. kafka.controller:type=KafkaController,name=MetadataWriteOffsesMetadataLastCommittedRecordOffset
The active controller will report the offset of the latest writethat last committed offset it consumed. Inactive controllers will always report 0the same value in MetadataLastAppliedRecordOffset.
3. kafka.controller:type=KafkaController,name=MetadataLastCommittedTimestampMetadataLastAppliedRecordTimestamp
   Reports the append time of the last committed applied record batch.
4. kafka.controller:type=KafkaController,name=MetadataLastCommittedLagMsMetadataLastAppliedRecordLagMs
   Reports the difference between the local time and the append time of the last committed applied record batch.

Broker

1. kafka.server:type=broker-metadata-metrics,name=last-committedapplied-record-offset 
   Reports the offset of the last record consumed by the broker.
2. kafka.server:type=broker-metadata-metrics,name=last-committedapplied-record-timestamp 
3. kafka.server:type=broker-metadata-metrics,name=last-committedapplied-record-lag-ms
4. kafka.server:type=broker-metadata-metrics,name=pending-record-processing-time-us-avg
   Reports the average amount of time it took for the broker to process all pending committed records when there are pending records in the cluster metadata partition. The time unit for this metric is microseconds.
5. kafka.server:type=broker-metadata-metrics,name=pending-record-processing-time-us-max
   Reports the maximum amount of time it took for the broker to process all pending committed records when there are pending records in the cluster metadata partition. The time unit for this metric is microseconds.
6. kafka.server:type=broker-metadata-metrics,name=record-batch-size-byte-avg
   Reports the average byte size of the record batches in the cluster metadata partition.
7. kafka.server:type=broker-metadata-metrics,name=record-batch-size-byte-max
   Reports the maximum byte size of the record batches in the cluster metadata partition.

Configuration

1. metadata.monitormax.writeidle.interval.ms - The interval between writing NoopRecord NoOpRecord to the cluster metadata log. The default for this value is 500 ms.

Proposed Changes

Active Controller

The active controller will increase the LEO and high-watermark by periodically writing a noop no-op record (NoopRecordNoOpRecord) to the metadata log. The active controller will write this new record only if the IBP and metadata.version supports this feature. See the backward compatibility section for details.

The implementation will only append the NoopRecordNoOpRecord, if the LEO wasn’t already advanced in the defined period.

...

In both the controller and broker, the metadata replaying code will be extended to ignore NoopRecordNoOpRecord.

Compatibility, Deprecation, and Migration Plan

The IBP and metadata.version will be bumped. This feature and record will only be produced if the active controller is at the expected version or greater.

Users must use the same software version of DumpLogSegment associated with the server node when reading the metadata cluster log segments.

Rejected Alternatives

Control Records

Instead of using the NoopRecord NoOpRecord metadata record. We could have added a control record in the KRaft layer. This solution has two problems.

...