Table of Contents |
---|
Status
Current state: Under DiscussionApproved
Discussion thread: here
JIRA: KAFKA-13883
...
Public Interfaces
Cluster Metadata Records
...
NoOpRecord
Add a new record to periodically advancement of the LEO and high-watermark. Controller or broker state will not change when applying this record. This record will not be included in the cluster metadata snapshot.
Code Block |
---|
{ “apiKey”: TBD, “type”: “metadata”, “name”: “NoopRecord”“NoOpRecord”, “validVersions”: “0”, “flexibleVersions”: “0+”, “fields”: [] } |
...
kafka.controller:type=KafkaController,name=MetadataLastCommittedOffsetMetadataLastAppliedRecordOffset
Reports the offset of the last committed offset record consumed by the controller. For the active controller this may include uncommitted records. For the inactive controller this always includes committed records.kafka.controller:type=KafkaController,name=MetadataWriteOffsesMetadataLastCommittedRecordOffset
The active controller will report the offset of the latest writethat last committed offset it consumed. Inactive controllers will always report 0the same value in MetadataLastAppliedRecordOffset.kafka.controller:type=KafkaController,name=MetadataLastCommittedTimestampMetadataLastAppliedRecordTimestamp
Reports the append time of the last committed applied record batch.kafka.controller:type=KafkaController,name=MetadataLastCommittedLagMsMetadataLastAppliedRecordLagMs
Reports the difference between the local time and the append time of the last committed applied record batch.
Broker
kafka.server:type=broker-metadata-metrics,name=last-applied-record-offset
Reports the offset of the last record consumed by the broker.kafka.server.metadata:type=BrokerMetadataPublisherbroker-metadata-metrics,name=MetadataLastCommittedOffsetlast-applied-record-timestamp
kafka.server:type=broker-metadata-metrics,name=last-applied-record-lag-ms
kafka.server:type=broker-metadata-metrics,name=pending-record-processing-time-us-avg
Reports the average amount of time it took for the broker to process all pending committed records when there are pending records in the cluster metadata partition. The time unit for this metric is microseconds.kafka.server:type=BrokerMetadataPublisher,name=MetadataLastCommittedTimestamp
kafka.server.metadata:type=BrokerMetadataPublisher,name=MetadataLastCommittedLagMs
Configuration
broker-metadata-metrics,name=pending-record-processing-time-us-max
Reports the maximum amount of time it took for the broker to process all pending committed records when there are pending records in the cluster metadata partition. The time unit for this metric is microseconds.kafka.server:type=broker-metadata-metrics,name=record-batch-size-byte-avg
Reports the average byte size of the record batches in the cluster metadata partition.kafka.server:type=broker-metadata-metrics,name=record-batch-size-byte-max
Reports the maximum byte size of the record batches in the cluster metadata partition.
Configuration
- metadata.max.idle.interval.ms - The interval between writing NoOpRecord to the cluster metadata log. The default for this value is 500 msconfluent.metadata.monitor.write.ms - Frequency for writing NoopRecord to the cluster metadata log.
Proposed Changes
Active Controller
The active controller will increase the LEO and high-watermark by periodically writing a noop no-op record (NoopRecordNoOpRecord) to the metadata log. The active controller will write this new record only if the IBP and metadata.version supports this feature. See the backward compatibility section for details.
The implementation will only append the NoopRecordNoOpRecord, if the LEO wasn’t already advanced in the defined period.
...
In both the controller and broker, the metadata replaying code will be extended to ignore NoopRecordNoOpRecord.
Compatibility, Deprecation, and Migration Plan
The IBP and metadata.version will be bumped. This feature and record will only be produced if the active controller is at the expected version or greater.
Users must use the same software version of DumpLogSegment
associated with the server node when reading the metadata cluster log segments.
Rejected Alternatives
Control Records
Instead of using the NoopRecord NoOpRecord metadata record. We could have added a control record in the KRaft layer. This solution has two problems.
...