THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
Proposed Changes
High-level design
RemoteLogManager (RLM) is a new component which
...
Ry = Remote log end offset Rx = Remote log start offset
Lz >= Ly and Ly >= Lx and Ly >= Ry and Ry >= Rx
Manage Remote Log Segments
...
For follower partitions, it maintains metadata cache by subscribing to the respective remote log metadata topic partitions. Whenever a topic partition is reassigned to a new broker and RLMM on that broker is not subscribed to the respective remote log metadata topic partition then it will subscribe to the respective remote log metadata topic partition and adds all the entries to the cache. So, in the worst case, RLMM on a broker may be consuming from most of the remote log metadata topic partitions. This requires the cache to be based on disk storage like RocksDB to avoid a high memory footprint on a broker. This will allow us to commit offsets of the partitions that are already read. Committed offsets can be stored in a local file to avoid reading the messages again when a broker is restarted.
Remote Log Indexes
| Anchor | ||||
|---|---|---|---|---|
|
...