...

As per the Tiered Storage feature introduced in KIP-405, the follower fetch protocol was modified for topics enabled with tiered storage. For such topics, the empty follower finds the offset and leader epoch, up to which the auxiliary state (i.e. leader epoch sequence, producer snapshot state) needs to be built from the leader. The follower then starts fetching the data from the leader starting from that offset. In the KIP, it was decided to use the local-log-start-offset as the offset for this purpose. Instead of using the local-log-start-offset as the starting offset from where the follower starts replicating the data from the leader, we could also use the last-tiered-offset as the starting offset (technically we will use the next offset for last-tiered-offset).

This offset is the last offset that has been uploaded to the remote storage. With this option, the follower will only need to replicate a small number of segments from the leader (in regular cases, . This is because in most cases where there is no lag and , most of the inactive log segments are already available on the remote storage). If this follower does become a leader in the future, it can still serve older segments because they are already on remote storage. Because the follower needs to copy only a small number of segments from the leader, it can quickly catch up with the leader and join the ISR list for the topic partition. In practice, the amount of data the follower needs to replicate from the leader can be as low as 10-15%. Therefore the time for the new follower to join the ISR list is also 10-15% compared to before. What this means, implies is that with this strategy an empty broker can quickly join the ISR list for the topic partitions that are assigned to it, improving the availability of the Kafka cluster.

A drawback of using the last-tiered-offset is that this new follower would possess only a limited number of locally stored segments. Should it ascend to the role of leader, there is a risk of needing to fetch these segments from the remote storage, potentially impacting broker performance. There can be several ways to prevent such followers from transitioning to a leader role, such as having strict leader election criteria, wherein followers with limited locally stored segments are not allowed to become a leader, or having slightly relaxed criteria wherein other followers are preferred for leadership over this follower. Incorporating these preventing measures alongside the proposed utilization of last-tiered-offset can effectively balance the advantage of rapid follower synchronization and mitigate potential performance impacts on the broker, thereby optimizing the overall efficiency and reliability.

In this KIP, we will focus only on the follower fetch protocol using the last-tiered-offset, which will be helpful in quickly converting empty followers to in-sync followers. This can drastically reduce the time taken to run cluster rebalances. Similarly, it can drastically shorten the time to replace a faulty broker.

Public Interfaces

Broker Configuration options

...