Page History

BookKeeper auto recovery discussed in BOOKKEEPER-237 JIRA and already implemented many sub-tasks in it.
We have to discuss about Fsck feature. Edit this page

...

When any Bookie goes down in the BookKeeper cluster, there is no way to recover the lost data from that Bookie server. For example, if we have 2 replicas for a ledger in BK cluster, and a node goes down from it, we will be running the cluster with single replica. Running clusters with single or no replicas will be a risk, as nodes may fail in general. To avoid such situations, we need a mechanism for recovering the data to new bookies for meeting the enough replica criteria (quorum size) and it is called as Auto-Recovery in BookKeeper.

...

•  Auditor
•  ReplicationWorker

AuditorPeer AutoRecoveryMain is an Auto-recovery node, which internally initializes and starts Auditor and ReplicationWorker threads. So, each Auto-recovery node will have two threads running.

This Auto-recovery node has to be started in each Bookie machine. All recovery nodes will participate in leader election and one Auditor may become as the leader and others will just watch the elected auditor failure to participate again in next election.

Auditor:

Once the Auditor thread is started, the auditor elector will go for the election to win the auditing job for Bookie cluster. Here, auditing job would be that, it has to detect the under-replicated ledgers in the cluster due to Bookie failures.

...

How ReplicationWorker handled this data loss scenario?
Scenario: The last fragment of the ledger is in under replicated state; replication worker replicates it and updates the ledger metadata with local Bookies address. Immediately, the failed Bookie started and running. Now the client resumed for adding some more entries, and it can continue with adding entries with the old Bookie. But ReplicationWorker already change the metadata for that fragment with local Bookie. That means, that client unnecessarily adding the entries to the old bookie whose address is already removed from fragment ensemble. So, this can create data loss if other bookie goes down and even though old Bookie is running fine.
To prevent this situation, ReplicationWorker will postpone the replications by adding that ledger to the pending replications. Pending ReplicationWorker will check the timeout of the ledgers which are there in pending replications. This timeout is configurable. Once the timeout happens, pending ReplicationWorker if the last fragment of the ledger is in open state. In such case it will just schedule a timer task for that ledger for delaying replication for such ledgers. That timer task scheduling period is configurable and default value is 30000ms. Once the timer fired, it will force fence the ledger if it is still in open state and will inform the replication worker for the
replicationrelease the ledger lock.So, that will trigger rereplication automatically as RW will loop to get the under replicated ledgers. So, any under-replicated last fragment ledger will not be kept open for long time if the client is idle and not reforming ensemble for long (more than pending replication timeout.)

...