THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
This KIP proposes to add the possibility for RemoteStorageManager.copyLogSegmentData method to return optional custom metadata after a segment has been uploaded. This piece will be stored and propagated along with the standard metadata. The custom metadata will be a opaque byte array to the rest of the system and . It should be written serialized, deserialized, and interpreted only by the remote storage plugin only.
Allowing remote storage plugins to write custom metadata of arbitrary size is not good for the stability of the cluster. The broker should have a configurable limit for their size. If a piece of custom metadata exceeds the limit, the execution of RLMTask.copyLogSegmentsToRemote should be interrupted with an error message. The default value for the limit should be 10 KiB, which is enough to store 10 1024-byte ANSI strings (minus some bytes for e.g. the map structure), which is quite sufficient for the two use cases from the Motivation part and on the other hand is small enough to not be disruptive.
...
Same as for RemoteLogSegmentMetadataRecord.
Configuration Keys
| Key Name | Description | Valid Values | Default Value |
|---|---|---|---|
remote.log.metadata.custom.metadata.max.size | The maximum size of custom metadata that the broker should accept from a remote storage plugin. If custom metadata exceeds this limit, the whole segment metadata will not be stored and the remote copying task for this topic-partition will stop with an error. | 0..Integer.MAX_VALUE |
(10 KiB) |
Compatibility, Deprecation, and Migration Plan
...