Versions Compared

Old Version 63

changes.mady.by.user doug chang

Saved on

compared with

New Version Current

changes.mady.by.user Olaf Flebbe

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Introduction

Installing Bigtop Hadoop distribution artifacts lets you have an up and running Hadoop cluster complete with
various Hadoop ecosystem projects in just a few minutes. Be it a single node pseudo-distributed
configuration, or a fully distributed cluster, just make sure you install the packages, install the JDK,
format the namenode and have fun! If Bigtop is not supported on your OS, you can install one of the supported 64-bit OSes
on a virtual machine. There is known issues with 32-bit OSes.

Table of Contents

Getting the packages onto your box

CentOS 5, CentOS 6, Fedora 15, RHEL5, RHEL6

...

  1. Make sure to grab the repo file:

    No Format
  1. wget -O /etc/yum.repos.d/bigtop.repo http://www.apache.org/dist

...

  1. /bigtop/stable/repos/[centos5|centos6|fedora15|

...

  1. fedora16]/bigtop.repo

...

  2. This step is optional, 
...
  1.  but 
...
  1.  recommended: 
...
  1.  enable 
...
  1.  the 
...
  1.  mirror 
...
  1.  that 
...
  1.  is 
...
  1.  closest 
...
  1.  to 
...
  1.  you 
...
  1.  (uncomment 
...
  1.  one 
...
  1.  and 
...
  1.  only 
...
  1.  one 
...
  1.  of 
...
  1.  the 
...
  1.  baseurl 
...
  1.  lines 
...
  1. and 
...
  1.  remove
...
  1.  the 
...
  1.  mirrorlist 
...
  1.  line). 
...
  1.  If 
...
  1.  the 
...
  1.  downloads 
...
  1.  are 
...
  1.  too 
...
  1.  slow, 
...
  1.  try 
...
  1.  another 
...
  1.  mirror
...
  1.  No Format
...
  1. sudo vi /etc/yum.repos.d/bigtop.repo

...
  2. Note: Since 0.3.0 is currently available only from the archives use:
     No Format
    baseurl=http://archive.apache.org/dist/incubator/bigtop/bigtop-0.3.0-incubating/repos/[centos5|centos6|fedora15|fedora16]/

  3. Browse through the artifacts
     No Format
    yum search hadoop

...
  1. yum --disablerepo "*" --enablerepo "bigtop-0.3.0-incubating" list available

  2. Install the full Hadoop stack (or parts of it)
     No Format
    sudo yum install hadoop\* flume\* mahout\* oozie\* whirr\* hbase\*

...
SLES 11, 
...
 OpenSUSE
...
  1. Make 
...
  1.  sure 
...
  1.  to 
...
  1.  grab 
...
  1.  the 
...
  1.  repo 
...
  1.  file:
...
  1.  No Format
...
  1. wget -O  http://www.apache.org/dist/incubator/bigtop/stable/repos/suse/bigtop.repo
mv bigtop.repo  /etc/zypp/repos.d/bigtop.repo

...
  2. Enable the mirror that is closest to you (uncomment one and only one of the baseurl lines). If the downloads are too slow, try another mirror
     No Format
    As root:  vi /etc/zypp/repos.d/bigtop.repo

...
  2. Browse through the artifacts
     No Format
    zypper search hadoop

...
  2. Install the full Hadoop stack (or parts of it)
     No Format
    sudo zypper install hadoop\* flume\* mahout\* oozie\* whirr\*

...
Ubuntu
  1. Install the Apache Bigtop GPG key
     No Format
    wget -O- http://www.apache.org/dist/incubator/bigtop/bigtop-0.
...
  1. 3.0-incubating/repos/GPG-KEY-bigtop | sudo apt-key add -

...
  2. Make sure to grab the repo file:
     No Format
    sudo wget -O /etc/apt/sources.list.d/bigtop.list http://www.apache.org/dist/incubator/bigtop/bigtop-0.
...
  1. 3.0-incubating/repos/ubuntu/bigtop.list

...
  2. Enable the mirror that is closest to you (uncomment one and only one pair of deb/deb-src 
...
  1.  lines). 
...
  1.  If 
...
  1.  the 
...
  1.  downloads 
...
  1.  are 
...
  1.  too 
...
  1.  slow, 
...
  1.  try 
...
  1.  another 
...
  1.  mirror
...
  1.  No Format
...
  1. sudo vi /etc/apt/sources.list.d/bigtop.list

...
  2. Update the apt cache
     No Format
    sudo apt-get update

...
  2. Browse through the artifacts
     No Format
    apt-cache search hadoop

...
  2. Make sure that you have the latest JDK installed on your system as well. You can either get it from the official Oracle website (http://www.oracle.com/technetwork/java/javase/downloads/jdk-6u29-download-513648.html
...
  1. ) 
...
  1.  or 
...
  1.  follow 
...
  1.  the 
...
  1.  advice 
...
  1.  given 
...
  1.  by 
...
  1.  your 
...
  1.  Linux 
...
  1.  distribution. 
...
  1.  If 
...
  1.  your 
...
  1.  JDK 
...
  1.  is 
...
  1.  installed 
...
  1.  in 
...
  1.  a 
...
  1.  non-standard 
...
  1.  location, 
...
  1.  make 
...
  1.  sure 
...
  1.  to 
...
  1.  add 
...
  1.  the 
...
  1.  line 
...
  1.  below 
...
  1.  to 
...
  1.  the 
...
  1.  /etc/default/hadoop 
...
  1.  file
...
  1.  No Format
...
  1. export JAVA_HOME=XXXX

...
  2. Install the full Hadoop stack (or parts of it)
     No Format
    sudo apt-get install hadoop\* flume-* mahout\* oozie\* whirr-*

...
Running Hadoop
After installing Hadoop packages onto your Linux box, make sure that:
  1. You have the latest JDK installed on your system as well. You can either get it from the official Oracle website (http://www.oracle.com/technetwork/java/javase/downloads/jdk-6u29-download-513648.html
...
  1. ) 
...
  1.  or 
...
  1.  follow 
...
  1.  the 
...
  1.  advice 
...
  1.  given 
...
  1.  by 
...
  1.  your 
...
  1.  Linux 
...
  1.  distribution 
...
  1.  (e.g. 
...
  1.  some 
...
  1.  Debian 
...
  1.  based 
...
  1.  Linux 
...
  1.  distributions 
...
  1.  have 
...
  1.  JDK 
...
  1.  packaged 
...
  1.  as 
...
  1.  part 
...
  1.  of 
...
  1.  their 
...
  1.  extended 
...
  1.  set 
...
  1.  of 
...
  1.  packages). 
...
  1.  If 
...
  1.  your 
...
  1.  JDK 
...
  1.  is 
...
  1.  installed 
...
  1.  in 
...
  1.  a 
...
  1.  non-standard 
...
  1.  location, 
...
  1.  make 
...
  1.  sure 
...
  1.  to 
...
  1.  add 
...
  1.  the 
...
  1.  line 
...
  1.  below 
...
  1.  to 
...
  1.  the 
...
  1.  /etc/default/hadoop 
...
  1.  file
...
  1.  No Format
...
  1. export JAVA_HOME=XXXX

...
  2. Format the namenode
     No Format
    sudo -u hdfs hadoop namenode -format

...
  2. Start the necessary Hadoop services. E.g. 
...
  1.  for 
...
  1.  the 
...
  1.  pseudo 
...
  1.  distributed 
...
  1.  Hadoop 
...
  1.  installation 
...
  1.  you 
...
  1.  can 
...
  1.  simply 
...
  1.  do:
...
  1.  No Format
...
  1. for i in hadoop-namenode hadoop-datanode hadoop-jobtracker hadoop-tasktracker ; do sudo service $i start ; done

...
  2. Once your basic cluster is up and running it is a good idea to create a home directory on the HDFS:
     No Format
    sudo -u hdfs hadoop fs -mkdir /user/$USER
sudo -u hdfs hadoop fs -chown $USER /user/$USER

...
  2. Enjoy your cluster
     No Format
    hadoop fs -lsr /
hadoop jar /usr/lib/hadoop/hadoop-examples.jar pi 10 1000

...
  2. If you are using Amazon AWS it is important the IP address in /etc/hostname 
...
  1.  matches 
...
  1.  the 
...
  1.  Private 
...
  1.  IP 
...
  1.  Address 
...
  1.  in 
...
  1.  the 
...
  1.  AWS 
...
  1.  Management 
...
  1.  Console. 
...
  1.  If 
...
  1.  the 
...
  1.  addresses 
...
  1.  do 
...
  1.  not 
...
  1.  match 
...
  1.  Map 
...
  1.  Reduce 
...
  1.  programs 
...
  1.  will 
...
  1.  not 
...
  1. complete. 
     Image Added
     No Format
    ubuntu@ip-10-224-113-68:~$ cat /etc/hostname
ip-10-224-113-68

...
  2. If the IP address in /etc/hostname 
...
  1.  does 
...
  1.  not 
...
  1.  match 
...
  1.  then 
...
  1.  open 
...
  1.  the 
...
  1.  hostname 
...
  1.  file 
...
  1.  in 
...
  1.  a 
...
  1.  text 
...
  1.  editor, 
...
  1.  change 
...
  1.  and 
...
  1. reboot
Running Hadoop Components
 
 HTML
<h3>
Here are <a href="https://cwiki.apache.org/confluence/display/BIGTOP/Running+various+Bigtop+components" target="_blank">step-by-step instructions on running Hadoop Components!</a>
</h3>

 
One of the advantages of Bigtop is the ease of installation of the different Hadoop Components without having to hunt for a specific Hadoop Component distribution and matching it with a specific Hadoop version.
Please visit the link above to run some easy examples from the Bigtop distribution !
Provided at the link above are examples to run Hadoop 1.0.1 and nine other components from the Hadoop ecosystem (hive/hbase/zookeeper/pig/sqoop/oozie/mahout/whirr and flume).
See the
 HTML
<a href="https://svn.apache.org/repos/asf/incubator/bigtop/trunk/bigtop.mk" target="_blank">Bigtop Make File</a>
for a list of Hadoop components , officially available from the Bigtop distribution.
 
Where to go from here
It is highly recommended that you read documentation provided by the Hadoop project itself
  1. https://hadoop.apache.org/common/docs/
...
  1. r1.
...
  1. 0.
...
  1. 1/ for Bigtop 0.3
  2. or, http://hadoop.apache.org/common/docs/
...
  1. r0.20.
...
  1. 205.0/
...
  1. ) Bigtop 
...
  1.  0.
...
  1. 2
and 
...
 that 
...
 you 
...
 browse 
...
 through 
...
 the 
...
 Puppet 
...
 deployment 
...
 code 
...
 that 
...
 is 
...
 shipped 
...
 as 
...
 part 
...
 of 
...
 the 
...
 Bigtop 
...
 release 
...
 (bigtop-deploy/puppet/modules, 
...
 bigtop-deploy/puppet/manifests).
...