Versions Compared

Old Version 16

changes.mady.by.user Unknown User (showuon)

Saved on

compared with

New Version Current

changes.mady.by.user Unknown User (showuon)

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

1. Cluster B creates a mirror and starts mirroring from the source cluster A. At this point, because the source cluster stores nothing about the LMLE for this partition, -1 will be returned. The "-1" means the source cluster (A) doesn't recognize any leader epoch in destination cluster (B), so all the logs should be truncated. Later, the cluster B mirrors batches from cluster A, including the leader epoch in the batches.

2. Revere mirroring. Cluster B becomes the source cluster. And before cluster B stops the mirroring from A, it stores the LMLE in the internal topic, which is 1 in this case. When cluster A mirrors from cluster B, it will first query the LMLE from cluster B, and then truncate every record beyond this leader epoch. That is, for cluster A and B, leader epochs for [0, 1] are in sync.

Also, in the cluster B, when it stops the mirror from A, it also bumps the leader epoch to a leader epoch > 1. In this case, it is 2, to make sure the leader epoch is increasing. 

Later, the cluster A mirrors batches from cluster B, including the leader epoch in the batches.

3. Reverse mirroring again. Cluster A stores LMLE 3 in the internal topic, and cluster B queries this LMLE and truncate records beyond leader epoch 3. Besides, the leader epoch in cluster A bumps to 4. Later, the cluster B mirrors batches from cluster A, including the leader epoch in the batches.


Please note that after LMLE truncation, it doesn't mean the log between 2 clusters are in sync. It only means the leader epoch history is in sync. The leader epoch history (i.e. the leader-epoch-checkpoint file in the partition directory) contains the entry for each [leader epoch, start offset]. However, the log might still not converge, yet. In the example below, after cluster B truncates LMLE 3, the offset 4 (in leader epoch 3) still exists, which is inconsistent with the source cluster. In this case, the second log truncation will be triggered by the replication protocol by the last fetched leader epoch comparison. So the offset 4 will be truncated this time, and the logs in these 2 clusters are completely in sync.

...