DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
Terms:
Last Mirrored Leader Epoch(LMLE): This is the leader epoch that the source cluster recognizes in this destination cluster. Or we can say, below this leader epoch (inclusive), the source cluster and destination cluster is in sync in the perspective of leader epoch.
...
Please note that after LMLE truncation, it doesn't mean the log between 2 clusters are in sync. It only means the leader epoch history is in sync. The leader epoch history (i.e. the leader-epoch-checkpoint file in the partition directory) contains the entry for each [leader epoch, start offset]. However, the log might still not converge, yet. In the example below, after cluster B truncates LMLE 3, the offset 4 (in leader epoch 3) still exists, which is inconsistent with the source cluster. In this case, the second log truncation will be triggered by the replication protocol by the last fetched leader epoch comparison. So the offset 4 will be truncated this time, and the logs in these 2 clusters are completely in sync.
Log truncation in all ISRs
When the log truncation is processing before mirroring, the leader node will wait until all ISRs (and ISR size >= "min.insync.replicas" value) complete the truncation. This is to make sure all ISRs logs are in sync.
When unclean leader election comes into play
The unclean leader election means a replica that is not part of ISR becomes the leader. It will cause data loss because some committed records haven't replicated to this new leader replica, yet. With the LMLE design, it can resolve unclean leader election happened before mirroring. For example:
1. cluster B is mirroring from cluster A.
2. broker 0 in cluster A is down. No ISR is available, and unclean leader election triggered to a replica with only offset 0 in log.
3. New records append to the new leader in offset 1,2,3.
4. If cluster B continues to mirror from the cluster A, it’ll identify the epoch diverge and do log truncation to offset 0. But before the fetch request is sent, the whole cluster A is down and failover to cluster B. The cluster A will query the LMLE, and cluster B responds with 1. So records with leader epoch > 1 will be truncated.
...
Unresolvable cases
The example above resolves the unclean leader election before the reverse mirroring. However, the unclean leader election could happen anytime. And this might mess up the design above because the log truncation will wait only for all ISRs are completed. That means, once unclean leader election happens, the non-ISR might contain records in leader epoch beyond the LMLE. Thus the replication protocol cannot detect the log diverge and cause inconsistent data between these 2 clusters. For example:
1. Continue the step 3 above, the current log status is like this, after the unclean leader election.
2. Unclean leader election again in the cluster A, the old leader becomes the new leader now.
3. Failover to cluster B, and cluster A reverse mirroring from cluster B. These 2 clusters are in sync now.
4. Unclean leader election happened in cluster A, but no log truncation this time. Thus it causes log inconsistent.
