The BookKeeper journal manager is an implementation of the HDFS JournalManager interface. The JournalManager interface allows you to plug custom write ahead logging into the HDFS NameNode.
Setting up the BookKeeper Journal Manager
http://zookeeper.apache.org/bookkeeper/docs/r4.0.0/bookkeeperConfig.html covers the step necessary to set up a BookKeeper cluster. We recommend that you use at least 3 servers to ensure fault tolerance.
Once BookKeeper has been set up, you can configure the journal manager. Currently, JournalManagers are only available in trunk and 0.23.3 branches. BookKeeper JournalManager is available in trunk, but the most recent version is at https://github.com/ivankelly/hadoop-common/tree/BKJM-benching. All the changes in this branch are pending submission into trunk. For this guide we will use this branch.
...
This will configure the namenode with 1 local edits dir and 1 bookkeeper edits dir. The BookKeeperJournalManager connects to localhost:2181 as the ZooKeeper server. Note that this will not start the namenode is in high availability mode, but BookKeeperJournalManager can similarly be used as a shared edits dir in HA.
...
The web interface should now be available at http://localhost:50070/.
Benchmarks
We ran benchmarks on a cluster of machines with:
...
Key | Description |
---|---|
BookKeeper e3q2 | BookKeeper journal manager, with a an ensemble of 3 and quorum of 2, each Bookie using separate disk for journal and ledger. |
BookKeeper e6q2 | BookKeeper journal manager, with a an ensemble of 3 and quorum of 2, each Bookie using separate disk for journal and ledger. |
BookKeeper e3q2 1 disk | BookKeeper journal manager, with a an ensemble of 3 and quorum of 2, each Bookie using single disk for journal and ledger. |
BookKeeper e6q2 1 disk | BookKeeper journal manager, with a an ensemble of 3 and quorum of 2, each Bookie using single disk for journal and ledger. |
BookKeeper local (q1e1) | BookKeeper journal manager, with a single Bookie on the same machine as the namenode, using single disk for journal and ledger. |
File | File journal manager with a local file on a dedicated disk |
NFS | File journal manager with a file stored on enterprise NetApp Filer |