Table of Contents |
---|
Status
Current state: Under Discussion Accepted
Discussion thread: here
JIRA: here [Change the link from KAFKA-1 to your own ticket]
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
...
This KIP proposes to add the possibility for RemoteStorageManager.copyLogSegmentData
method to return optional custom metadata after a segment has been uploaded. This piece will be stored and propagated along with the standard metadata. The custom metadata will be a opaque byte array to the rest of the system and . It should be written serialized, deserialized, and interpreted only by the remote storage plugin only. The kafka-clients
library, which is distributed separately from Kafka and can be used as a plugin dependency, has convenient tools for binary serde.
Allowing remote storage plugins to write custom metadata of arbitrary size is not good for the stability of the cluster. The broker should have a configurable limit for their size. If a piece of custom metadata exceeds the limit, the execution of RLMTask.copyLogSegmentsToRemote
should be interrupted with an error message. The default value for the limit should be 10 KiB, which is enough to store 10 1024-byte ANSI strings (minus some bytes for e.g. the map structure), which is quite sufficient for the two use cases from the Motivation part and on the other hand 128 B, which is small enough to not be disruptive.
...
Code Block | ||||
---|---|---|---|---|
| ||||
{ "name": "CustomMetadata", "type": "bytes", "default": "null", "versions": "0+", "nullableVersions": "0+", "about": "Custom metadata." } |
RemoteLogSegmentMetadataUpdateRecord
Same as for RemoteLogSegmentMetadataRecord
.
RemoteLogSegmentMetadataSnapshotRecord
Same as for RemoteLogSegmentMetadataRecord
.
Configuration Keys
Key Name | Description | Valid Values | Default Value |
---|---|---|---|
remote.log.metadata.custom.metadata.max. |
bytes | The maximum size of custom metadata in bytes that the broker should accept from a remote storage plugin. If |
custom metadata exceeds this limit, the |
updated segment metadata will not be stored, the copied data will be attempted to delete, and the remote copying task for this topic-partition will stop with an error. | 0..Integer.MAX_VALUE |
10240
128 |
Compatibility, Deprecation, and Migration Plan
...