Page History

...

A critical scenario to consider is the failure of an entire Availability Zone, which would take down both the leader broker and its co-located S3E1Z bucket. In this rare event, the cluster remains available by electing a new leader in a different AZ. Once a new leader (e.g., in AZ2) is elected, it must ensure that the data previously written by the old leader is available for replication. To achieve this, the new leader republishes its local log segments to a new S3E1Z bucket located in its own AZ (AZ2). There are two strategies for this republishing process:

1. Comprehensive Approach: A safe method where the new leader re-uploads all log segments newer than the topic's local.retention.ms configuration. This ensures all potentially needed active data becomes available in the new location.
2. Optimized Approach: The new leader identifies and re-uploads only the log segments that have not yet been tiered to long-term storage. Followers needing older data can still fetch it directly from the long-term tiered storage, reducing the amount of data that needs to be immediately republished.

After the data is uploaded, the new leader publishes corresponding metadata to the __remote_wal_log_metadata topic. This results in duplicated metadata entries: one set from the old leader pointing to the inaccessible bucket in AZ1, and a new set from the new leader pointing to the new bucket in AZ2. Followers are designed to handle this situation gracefully. When a follower consumes metadata, it will first try to fetch data from the location specified. If it encounters the old metadata from the failed leader, the fetch will fail. The follower will then ignore this stale metadata and proceed to the next entry, which will be the valid metadata from the new leader. This ensures that replication continues correctly despite the temporary appearance of out-of-order metadata.

...