...
WARNING 1: We are only supporting Oracle JDK/JRE 6. Bigtop distribution may or may not run with OpenJDK. Please got to the Oracle's web site and download the JDK6 package for the Linux distro that you're using: http://www.oracle.com/technetwork/java/javasebusiness/downloads/java-archive-downloads-javase6-419409.html before installing Bigtop
...
- Make sure to grab the repo file:
No Format wget -O http://bigtop01.cloudera.org:8080/view/Releases/job/Bigtop-0.4.0/label=sles11/lastSuccessfulBuild/artifact/output/bigtop.repo sudo mv bigtop.repo /etc/zypp/repos.d/bigtop.repo
- Refresh zypper to start looking at the newly added repo
No Format sudo zypper refresh
- Browse through the artifacts
No Format zypper search mahout
- Install the full Hadoop stack (or parts of it)
No Format sudo zypper install hadoop\* flume\* mahout\* oozie\* whirr\*
...
- Install the Apache Bigtop GPG key
No Format wget -O- http://www.apache.org/dist/bigtop/bigtop-0.4.0-incubating/repos/GPG-KEY-bigtop | sudo apt-key add -
- Make sure to grab the repo file:
for Ubuntu Lucid:
for Ubuntu Precise:No Format sudo wget -O /etc/apt/sources.list.d/bigtop.list http://bigtop01.cloudera.org:8080/view/Releases/job/Bigtop-0.4.0/label=lucid/lastSuccessfulBuild/artifact/output/bigtop.list
No Format sudo wget -O /etc/apt/sources.list.d/bigtop.list http://bigtop01.cloudera.org:8080/view/Releases/job/Bigtop-0.4.0/label=precise/lastSuccessfulBuild/artifact/output/bigtop.list
- Update the apt cache
No Format sudo apt-get update
- Browse through the artifacts
No Format apt-cache search mahout
- Install bigtop-utils
No Format sudo apt-get install bigtop-utils
- Make sure that you have the latest JDK installed on your system as well. You can either get it from the official Oracle website (http://www.oracle.com/technetwork/java/javase/downloads/jdk-6u29-download-513648.html) or follow the advice given by your Linux distribution. If your JDK is installed in a non-standard location, make sure to add the line below to the /etc/default/bigtop-utils file
No Format export JAVA_HOME=XXXX
- Install the full Hadoop stack (or parts of it)
No Format sudo apt-get install hadoop\* flume-* mahout\* oozie\* whirr-*
...
- You have the latest JDK installed on your system as well. You can either get it from the official Oracle website (http://www.oracle.com/technetwork/java/javase/downloads/jdk-6u29-download-513648.html) or follow the advice given by your Linux distribution (e.g. some Debian based Linux distributions have JDK packaged as part of their extended set of packages). If your JDK is installed in a non-standard location, make sure to add the line below to the /etc/default/hadoop bigtop-utils file
No Format export JAVA_HOME=XXXX
- Format the namenode
No Format sudo /etc/init.d/hadoop-hdfs-namenode init
- Start the necessary Hadoop services. E.g. for the pseudo distributed Hadoop installation you can simply do:
No Format for i in hadoop-hdfs-namenode hadoop-hdfs-datanode ; do sudo service $i start ; done
- Make sure to create a sub-directory structure in HDFS before running any daemons:
No Format sudo -u hdfs hadoop fs -mkdir /user/$USER sudo -u hdfs hadoop fs -chown $USER:$USER /user/$USER sudo -u hdfs hadoop fs -mkdir /tmp sudo -u hdfs hadoop fs -chmod -R 1777 /tmp sudo -u hdfs hadoop fs -mkdir /var/log/hadoop-yarn sudo -u hdfs hadoop fs -chown yarn:mapred /var/log/hadoop-yarn sudo -u hdfs hadoop fs -mkdir /user/history sudo -u hdfs hadoop fs -chown mapred:mapred /user/history sudo -u hdfs hadoop fs -chmod 770 /user/history sudo -u hdfs hadoop fs -mkdir /tmp/hadoop-yarn/staging sudo -u hdfs hadoop fs -chmod -R 1777 /tmp/hadoop-yarn/staging sudo -u hdfs hadoop fs -mkdir /tmp/hadoop-yarn/staging/history/done_intermediate sudo -u hdfs hadoop fs -chmod -R 1777 /tmp/hadoop-yarn/staging/history/done_intermediate sudo -u hdfs hadoop fs -chown -R mapred:mapred /tmp/hadoop-yarn/staging
- Now start YARN daemons:
No Format sudo service hadoop-yarn-resourcemanager start sudo service hadoop-yarn-nodemanager start
- Enjoy your cluster
No Format hadoop fs -lsr / hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples*.jar pi 10 1000
- If you are using Amazon AWS it is important the IP address in /etc/hostname matches the Private IP Address in the AWS Management Console. If the addresses do not match Map Reduce programs will not complete.
No Format ubuntu@ip-10-224-113-68:~$ cat /etc/hostname ip-10-224-113-68
- If the IP address in /etc/hostname does not match then open the hostname file in a text editor, change and reboot
...
One of the advantages of Bigtop is the ease of installation of the different Hadoop Components without having to hunt for a specific Hadoop Component distribution and matching it with a specific Hadoop version.
Please visit the link above to run some easy examples from the Bigtop distribution !
Provided at the link above are examples to run Hadoop 1.0.1 and nine other components from the Hadoop ecosystem (hive/hbase/zookeeper/pig/sqoop/oozie/mahout/whirr and flume).
See the
HTML |
---|
<a href="https://svn.apache.org/repos/asf/incubator/bigtop/trunk/bigtop.mk" target="_blank">Bigtop Make File</a> |
...