BookKeeper

BookKeeper is a system to reliably log streams of records. It is designed to store write ahead logs, such as those found in database or database like applications. In fact, the Hadoop NameNode inspired BookKeeper. The NameNode logs changes to the in-memory namespace data structures to the local disk before they are applied in memory. However logging the changes locally means that if the NameNode fails the log will be inaccessible. We found that by using BookKeeper, the NameNode can log to distributed storage devices in a way that yields higher availability and performance. Although it was designed for the NameNode, BookKeeper can be used for any application that needs strong durability guarantees with high performance and has a single writer.

...

The BookKeeper data model is a flat namespace of ledgers identified by a long. The ledgers are append only and writable by a single client. The basic operations of BookKeeper are: createLedger to create a new ledger available for writing, openLedger to read from an existing ledger, addEntry, removeEntry, and closeLedger. Once a ledger is closed it becomes read-only.

Documentation:

• Latest documentation

What is going on:

• Bookie recovery (BookieRecoveryPage)
• Bookie registration and failure detection (BookieRegPage)
• Ledger deletion (LedgerDeletionPage)
• Performance numbers(BookKeeperPerfPage)