THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
This was done in Centos 6.5
Note for alternative file systems (i.e. HCFS implementations like S3FileSystem, the GlusterFileSystem, and so on)
- you will want to disable HDFS specific modules and set up your file system by hand.
- The other YARN related puppet modules will still work perfectly if you've set up your file system correctly. For specifics consult your file system provider and/or the bigtop mailing list.
- One simple way to do this, is set up bigtop hadoop with hdfs, make sure it runs, and then disable and yum remove hdfs and swap in your file system underneath .
Okay ! So Lets get started setting up your hadoop cluster .
0) Install all the basics in case you have a super raw machine. Most (or some of these) are probably there.
...
To bring the cluster up for the first time (disclaimer: independent execution of Puppet recipes on the cluster's nodes will automatically create HDFS structures and bring-up the services if all dependencies are satisfied, e..g configs are created, packages are installed, etc. If Puppet reports errors you might need to do the manual startup):
1) As root, run
...
| Code Block | ||
|---|---|---|
| ||
# /etc/init.d/hadoop-hdfs-namenode init (omit unless you want to |
...
start with nothing in your HDFS) |
...
# |
...
/etc/init.d/hadoop-hdfs-namenode start |
...
# |
...
/etc/init.d/hadoop-hdfs-datanode start |
...
# /usr/lib/hadoop/libexec/init-hdfs.sh (not needed after the first run) |
...
# |
...
/etc/init.d/hadoop-yarn-resourcemanager start |
...
# |
...
/etc/init.d/hadoop-yarn-proxyserver start |
...
# /etc/init.d/hadoop-yarn-nodemanager start |
on the master node.
2) On each of the slave nodes, run
...
| Code Block | ||
|---|---|---|
| ||
# /etc/init.d/hadoop-hdfs-datanode start |
...
# |
...
/etc/init.d/hadoop-yarn-nodemanager |
...
start |
To bring the cluster down cleanly:
1) On each of the slave nodes, run
| Code Block |
|---|
...
| ||
# /etc/init.d/hadoop-yarn-nodemanager stop |
...
# |
...
/etc/init.d/hadoop-hdfs-datanode stop |
2) On the master, run
| Code Block | ||
|---|---|---|
| ||
# |
...
/etc/init.d/hadoop-yarn-nodemanager stop |
...
# |
...
/etc/init.d/hadoop-yarn-proxyserver stop |
...
# |
...
/etc/init.d/hadoop-yarn-resourcemanager stop |
...
# |
...
/etc/init.d/hadoop-hdfs-datanode stop |
...
# |
...
/etc/init.d/hadoop-hdfs-namenode stop |
...