Page History

Ledger recovery alone does not prevent a violation of the persistence property. Consider the following scenario. We have an application with a ledger writer and a ledger reader waiting until the ledger writer closes the ledger to read. The leader reader, however, suspects incorrectly that the ledger writer has crashed and proceeds to open the ledger and recover it. The reader reads as many entries as it can and writes the identifier of the last confirmed entry to the metadata store. The ledger writer concurrently writes more entries to the ledger successfully, even though the ledger has been closed already.

This case violates the persistence property. To prevent such cases, we implement a fencing mechanism. Fencing prevents two clients from modifying a ledger concurrently, either by adding entries or closing it. To implement fencing, we embed into the recovery protocol a mechanism to fence off bookies. With this mechanism, clients notify the bookies in the ensemble of a ledger that it needs to close the ledger. Once a bookie receives such a notification, it marks its ledger fragment as fenced and errors out any request to add an entry to the ledger. To implement this feature, we use a recovery flag in messages from clients to bookies. All messages from a ledger reader have the flag on while recovering the ledger.

This flag guarantees that all bookies with which the ledger reader exchanges messages during recovery promise not to process new add requests without the flag for the ledger. Since during recovery a client needs to contact at least one bookie in every ack quorum, in each ack quorum there is at least one bookie that has been fenced off during recovery. Consequently, we guarantee that the ledger writer is not able to add more entries successfully once enough bookies have been fenced. To prevent ensemble changes during recovery, we first note in the ledger metadata that recovery has started.

We show an example in the attached figure. In the example, we have an ensemble of three bookies and we replicate entries in two bookies. The ledger writer adds successfully up to entry 11 and writes entry 12 partially. The reader initiates recovery by first obtaining the last confirmed add from each bookie, and starts from the next after the latest, which corresponds to entry 12. The reader reads entry 12 and ensures that it is replicated in a quorum by writing back entry 12. The next entry, entry 13, does not exist, since it is not present in any of the bookies that would have it, had it been written. The reader finally declares the ledger closed and writes to the metadata store, using CAS. When the ledger tries to complete the add of entry 12, the bookie rejects the request because it has been fenced.