...

Authors: Henry Cai, Thomas Thornton

Status

Current state: Under Discussion Withdrawn

Discussion thread: here 

JIRA: KAFKA-19225 

...

A critical scenario to consider is the failure of an entire Availability Zone, which would take down both the leader broker and its co-located S3E1Z bucket. In this rare event, the cluster remains available by electing a new leader in a different AZ. Once a new leader (e.g., in AZ2) is elected, it must ensure that the data previously written by the old leader is available for replication. To achieve this, the new leader republishes its local log segments to a new S3E1Z bucket located in its own AZ (AZ2). There are two strategies for this republishing process:

1. Comprehensive Approach: A safe method where the new leader re-uploads all log segments newer than the topic's local.retention.ms configuration. This ensures all potentially needed active data becomes available in the new location.
2. Optimized Approach: The new leader identifies and re-uploads only the log segments that have not yet been tiered to long-term storage. Followers needing older data can still fetch it directly from the long-term tiered storage, reducing the amount of data that needs to be immediately republished.

replica reconciliation takes place. This is similar to standard Kafka. If followers have more events than the high watermark, they will need to truncate them. Below is an example to illustrate this:

Suppose there is a remote WAL enabled topic with replication factor 3. The current leader of a partition is B1. B1 has received data up to offset 10, written data to S3E1Z up to offset 10, and published metadata up to offset 10. One follower, B2, has replicated up to offset 5, and follower B3, up to offset 7. Suppose B1 dies (e.g., AZ goes down), and B2 is elected as the new leader. B2 overwrites the LEO/high-watermark based on its own LEO, which is 5. B2 will tell its followers to reset their LEO to 5. So B3, which had an LEO of 7, will discard 2 messages and reset 5 to match B2, the new leader. In addition, B2 will also publish metadata DELETE events for any metadata entries it receives that are after its LEO (e.g., for messages 6-10), since they are no longer valid.

In the case that B3 was elected, with the higher end offset of offset 7. It will perform replica reconciliation and discard its data that is greater than the high watermark of 5. This prevents any sacrifices in availability, and is equivalent to standard KafkaAfter the data is uploaded, the new leader publishes corresponding metadata to the __remote_wal_log_metadata topic. This results in duplicated metadata entries: one set from the old leader pointing to the inaccessible bucket in AZ1, and a new set from the new leader pointing to the new bucket in AZ2. Followers are designed to handle this situation gracefully. When a follower consumes metadata, it will first try to fetch data from the location specified. If it encounters the old metadata from the failed leader, the fetch will fail. The follower will then ignore this stale metadata and proceed to the next entry, which will be the valid metadata from the new leader. This ensures that replication continues correctly despite the temporary appearance of out-of-order metadata.

Data Durability

In acks=all flow, we maintain the same data durability as today by requiring the data gets to the follower before we can acknowledge back to the producer;

...