Page History

...

Remoteoffsetindex file and remotetimestampindex file are similar with the existing .index file (offset index) and .timeindex file (timestamp index). The only difference is that they index the corresponding remotelogindex file instead of a log segment file.

Manage Remote Log Segments Info in Followers

The RLM of a follower broker will retrieve remote segments information from the remote storage. It periodically checks the remote storage to find out the new segments shipped by the leader.

When the follower discovers a new segment in remote storage, it will retrieve the index entries from remote storage and append them into its local remote log index file. The RemoteOffsetIndex file and RemoteTimestampIndex file are updated accordingly.

The leader may failed to ship segment data to remote storage on time. In such a situation, the follower has to keep its local segment files, even if the configured retention time is reached. The local segment files (and the corresponding index files) can only be deleted in the following 2 cases:

1. the follower received the corresponding segment data info from remote storage and updated its index files
2. the local files are already older than the configured remote retention time

After restarted or after assigned new topic-partitions, the RLM of a follower will retrieve remote log segment info from remote storage, and the fetcher threads will fetch local log segment data from the leader.

When a remote segment is deleted in the remote storage, the follower deletes the corresponding remote index files. Follower does not delete remote index files based on remote retention time. It always wait for the leader to delete the remote segments in remote storage first.

Public Interfaces

Compacted topics will not have remote storage support. 

...