Page History

Authors: Sriharsha Chintalapani, Suresh Srinivas, Ying Zheng, Satish Duggana

...

There is no change in the semantics of this API. It deletes records until the given offset if possible. This is equivalent to updating logStartOffset of the partition log with the given offset if it is greater than the current log-start-offset and it is less than or equal to high-watermark. If needed, it will clean remote logs asynchronously after updating the log-start-offset of the log.

...

For timestamps >= 0, it returns the first message offset whose timestamp is >= to the given timestamp in the request. That means it checks in remote log time indexes first, after which local log time indexes are checked. 

JBOD related changes

Currently, JBOD is supported by altering log dirs in two ways.

Fetch

This is changed as the fetch request can be for data available in remote storage. For consumers, it fetches from the remote storage as mentioned earlier. For Follower replicas, it sends back the local log start offset at which the data can be copied. This is covered in detail earlier. 

LeaderAndIsr

This is received by RLM to register for new leaders so that the data can be copied to the remote storage. RLMM will also register the respective metadata partitions for the leader/follower partitions if they are not yet subscribed.  

Stopreplica

RLM receives a callback and unassigns the partition for leader/follower task, If delete option is enabled then it tries to remove respective segments from RLMM. 

OffsetForLeaderEpoch

Look into leader epoch checkpoint cache. This is stored in tiered storage and it may be fetched by followers from tiered storage. 

JBOD related changes

Currently, JBOD is supported by altering log dirs in two ways.

• Altering to a different dir on the local broker

  • This is not supported in this KIP but we may plan this in future releases.

• Altering to a dir on

• Altering to a different dir on the local broker

  • This can be done by copying remote log metadata files to the respective new topic partition directories in ReplicaAlterLogDirsThread. This will be implemented in the future.

• Altering to a dir on a remote broker

  • This is equivalent to reassigning partitions to a different broker, which is already supported in this KIP as part of how followers behave with respect to remote tiered storage.

There are no changes with other protocol APIs because of tiered storage. 

RLM/RSM tasks and thread pools

Anchor
rlm-rsm-tasks rlm-rsm-tasks

...