THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
We will walk through several scenarios and show how this protocol change solves the problem.
Scenario 1: Leader change due to preferred leader election followed by immediate leader change.
Consider the very first scenario described in this document, where broker A becomes a leader with epoch 3. It then appends a new entry in the range (21, 30). To remind, the updated logs look like this:
| Code Block | ||
|---|---|---|
| ||
Broker A: 0: offsets [0, 10], leader epoch: 1 1: offsets [11, 20], leader epoch: 1 2: offsets: [21, 30], leader epoch: 3 Broker B: 0: offsets [0, 10], leader epoch: 1 1: offsets: [11, n], leader epoch: 2 |
On becoming follower, broker B sends OffsetForLeaderEpoch to broker A with leader_epoch 2. Broker A finds largest offset <= 2, and sends response {leader_epoch=1, offset = 21}. Broker B truncates all offsets for epochs > 1, in our example offsets [11, n], its LEO becomes 11. Since 21 > 11, broker B starts fetching from offset 11.
Scenario 2 (scenario 1 from KIP-101)
...
Here we show that scenarios fixed with KIP-101 will have the same behavior with this approach. Suppose we have brokers A and B. B is the leader. The following is there current state which also includes where their High Watermark is.
| Code Block |
|---|
Broker A: 0: offsets [0, 10], leader epoch: 1 1: offsets [11, 20], leader epoch: 1 HW = 11 Broker B: 0: offsets [0, 10], leader epoch: 1 1: offsets [11, 20], leader epoch: 1 HW = 21 |
Broker A restarts.
Broker A sends OffsetForLeaderEpoch request to broker B with leader_epoch = 1. Broker B responds with {leader_epoch 1, offset 21}. Broker A does not truncate.
Scenario 3: Fast leader fail over with unclean leader election.
This is the second scenario described in motivation. Here is the reminder:
| Info | ||
|---|---|---|
| ||
1. [LeaderEpoch0] Write a message to A (offset A:0), Stop broker A. Bring up broker B which becomes leader 2. [LeaderEpoch1] Write a message to B (offset B:0), Stop broker B. Bring up broker A which becomes leader 3. [LeaderEpoch2] Write a message to A (offset A:1), Stop broker A. Bring up broker B which becomes leader 4. [LeaderEpoch3] Write a message to B (offset B:1), 5. Bring up broker A. |
At step 5, broker A sends OffsetForLeaderEpoch to broker B with leader_epoch 2. Broker B responds with (leader_epoch 1, offset 1). Broker A sends another OffsetForLeaderEpoch to broker B with leader_epoch 0. Broker B responds with UNKNOWN_OFFSET_FOR_LEADER_EPOCH since it exhausted all epochs (in a more common case, there will be some epoch they both know about). Broker A will truncate to its HW which is offet 0 and starts fetching from offset 0.
...