Page History

...

         `RemoteStorageManager` is an interface to provide the lifecycle of remote log segments and indexes. We will provide a simple implementation of RSM to get a better understanding of the APIs. HDFS and S3 implementation are planned to be hosted in external repos and these will not be part of Apache Kafka repo. This is inline with the approach taken for Kafka connectors.

...

• receives callback events for leadership changes and stop/delete events of topic partitions on a broker.
• delegates copy, read, and delete of topic partition segments to a pluggable storage manager(viz RemoteStorageManager) implementation and maintains respective remote log segment metadata through RemoteLogMetadataManager.

We separated RemoteStorageManager and RemoteLogMetadataManager after going through the challenges we faced with eventually consistents stores like S3. You can see the details in the doc located here.

RLM creates tasks for each leader or follower topic partition:

...

RemoteLogMetadataManager implemented with an internal topic

Remote log metadata Metadata of partition’s remote log segments is stored in an internal topic called `__remote_log_metadata`. This topic can be created with default partitions count as 50. Users can configure the topic name and partitions count. 

...