Page History

...

It does not support compact topics.   

High-level design

Remote Log Manager RemoteLogManager (RLM) is a new component that keeps track of remote log segments 

• RLM component will keep track of topic-partition and its segments. It will delegate the copy and read of these segments to a pluggable storage manager implementation.
• RLM has two modes:
  • RLM Leader - In this mode, RLM is the leader for topic-partition, checks for rolled over LogSegments and copies it along with OffsetIndex their remote offset/time indexes to the remote tier. RLM creates an index file, called RemoteLogSegmentIndex, per topic-partition to track remote LogSegments. These indexes are described in detail in the next section. Additionally, the RLM leader also serves the read fetch requests for older data from the remote tier.
  • RLM Follower - In this mode, RLM keeps track of the segments and index files on remote tier and updates its RemoteLogSegmentIndex file per topic-partition. RLM follower does not serve reading old data from the remote tier.

...

         `RemoteStorageManager` is an interface that allows to plugin different remote storage implementations to provide the lifecycle of remote log segments. No implementation will be provioded as part of Apache Kafka binariesWe will provide a simple implementation of RSM to get a better understanding of the APIs. HDFS and S3 implementation are planned to be hosted in external repos and these will not be part of Apache Kafka repo. This is inline with the approach taken for Kafka connectors.

...