THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
RemoteLogManager is a new interface class added to the broker. It is responsible to copy the completed log segments to the remote storage and update RemoteOffsetIndex file. The default implementation of interface supports HDFS as remote storage. Additional remote storage support, such as S3 can be added later by providing other implementations using the configuration `Remote.log.manager.class`.
Only the broker that is the Leader for topic-partitions is allowed to copy to the remote storage.
...
Note: Early proposal. To be finalized during implementation.
| Code Block | ||
|---|---|---|
| ||
Trait RemoteStorageManager extends Configurable {
// Configure
def configure(Map<String, ?> configs)
// Copies LogSegment provided by the RLM
// Returns the RDIs of the remote data
// This method is used by the leader
def copyLogSegment(logSegment: LogSegment): (boolean, Seq[RemoteLogIndexEntry])
// List the remote log segment files of the specified topicPartition
// The RLM of a follower uses this method to find out the remote data
def listRemoteSegments(topicPartition: TopicPartition): Seq[RemoteLogSegmentInfo]
// Called by the RLM of a follower to retrieve RemoteLogIndex entries
// of the new remote log segment
def getRemoteLogIndexEntries(remoteLogSegment: RemoteLogSegmentInfo): Seq[RemoteLogIndexEntry])
// Deletes remote LogSegment file provided by the RLM
def deleteLogSegment(remoteLogSegment: RemoteLogSegmentInfo): boolean
// read topic partition data from remote storage,
// starting from the given offset.
def read(remoteLocation: RDI, maxBytes: Int, offset: Int): LogReadInfo
// stops all the threads and closes the instance.
def shutdown(): Unit
} |
...