Introduction

Installing Bigtop Hadoop distribution artifacts lets you have an up and running Hadoop cluster complete with
various Hadoop ecosystem projects in just a few minutes. Be it a single node pseudo-distributed
configuration, or a fully distributed cluster, just make sure you install the packages, install the JDK,
format the namenode and have fun!

Getting the packages onto your box

CentOS 5, CentOS 6, Fedora 15, RHEL5, RHEL6

1. Make sure to grab the repo file:

   sudo wget -O /etc/yum.repos.d/bigtop.repo http://www.apache.org/dist/incubator/bigtop/stable/repos/[centos5|centos6|fedora]/bigtop.repo
   

2. This step is optional, but recommended: enable the mirror that is closest to you (uncomment one and only one of the baseurl lines and remove the mirrorlist line):

   sudo vi /etc/yum.repos.d/bigtop.repo
   

3. Browse through the artifacts

   yum search hadoop
   

4. Install the full Hadoop stack (or parts of it)

   sudo yum install hadoop\* flume\* mahout\* oozie\* whirr\*
   

SLES 11, OpenSUSE

1. Make sure to grab the repo file:

   sudo wget -O /etc/zypp/repos.d/bigtop.repo http://www.apache.org/dist/incubator/bigtop/stable/repos/suse/bigtop.repo
   

2. Enable the mirror that is closest to you (uncomment one and only one of the baseurl lines)

   sudo vi /etc/zypp/repos.d/bigtop.repo
   

3. Browse through the artifacts

   zypper search hadoop
   

4. Install the full Hadoop stack (or parts of it)

   sudo zypper install hadoop\* flume\* mahout\* oozie\* whirr\*
   

Ubuntu

1. Install the Apache Bigtop GPG key

   wget -O- http://www.apache.org/dist/incubator/bigtop/stable/repos/GPG-KEY-bigtop | sudo apt-key add -
   

2. Make sure to grab the repo file:

   sudo wget -O /etc/apt/sources.list.d/bigtop.list http://www.apache.org/dist/incubator/bigtop/stable/repos/ubuntu/bigtop.list
   

3. Enable the mirror that is closest to you (uncomment one and only one pair of deb/deb-src lines)

   sudo vi /etc/apt/sources.list.d/bigtop.list
   

4. Update the apt cache

   sudo apt-get update
   

5. Browse through the artifacts

   apt-cache search hadoop
   

6. Install the full Hadoop stack (or parts of it)

   sudo apt-get install hadoop\* flume\* mahout\* oozie\* whirr\*
   

Running Hadoop

After installing Hadoop packages onto your Linux box, make sure that:

1. You have the latest JDK installed on your system as well. You can either get it from the official Oracle website (http://www.oracle.com/technetwork/java/javase/downloads/jdk-6u29-download-513648.html) or follow the advice given by your Linux distribution (e.g. some Debian based Linux distributions have JDK packaged as part of their extended set of packages). If your JDK is installed in a non-standard location,
   make sure to add

   export JAVA_HOME=XXXX
   

   to the /etc/default/hadoop file.
2. Format the namenode

   sudo -u hdfs hadoop namenode -format
   

3. Start the necessary Hadoop services. E.g. for the pseudo distributed Hadoop installation you can simply do:

   for i in hadoop-namenode hadoop-datanode hadoop-jobtracker hadoop-tasktracker ; do sudo service $i start ; done
   

4. Once your basic cluster is up and running it is a good idea to create a home directory on the HDFS:

   sudo -u hdfs hadoop fs -mkdir /user/$USER
   sudo -u hdfs hadoop fs -chown $USER /user/$USER
   

5. Enjoy your cluster

   hadoop fs -lsr /
   hadoop jar /usr/lib/hadoop/hadoop-examples.jar pi 10 1000
   

Where to go from here

It is highly recommended that you read documentation provided by the Hadoop project itself (http://hadoop.apache.org/common/docs/r0.20.205.0/) and that you browse through the Puppet deployment code that is shipped as part of the Bigtop release (bigtop-deploy/puppet/modules, bigtop-deploy/puppet/manifests).