Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Prompt user for hostname rather than making the user change the code, Welcome user, Check for CentOS 6, install git (make sure it's a dependency)

...

  • CentOS 6 machines running on VM/Docker/Bare Metals, etc

Deploy

Only ONE thing you need to do manually:

 

  • .
  • If using docker, don't run this on the docker host but on a CentOS 6 container

Deploy

Copy and paste the following text into a file called bigtop-deploy.sh, save the file, chmod it (e.g., chmod 0744 bigtop-deploy.sh), and run it (e.g., ./bigtop-deploy.sh).

Run bigtop-deploy.sh The deployment script(execute on EVERY node in your cluster):

Code Block
languagebash
# Welcome
RED='\033[0;31m'; BLUE='\033[0;34m'; DEFAULTC='\033[0m'
printf "\nWelcome to the ${RED}Bigtop${DEFAULTC} installer\n\n"


# Check for CentOS 6
rel=$(cat /etc/redhat-release 2> /dev/null)
if [[ -n "$rel" && ${rel:15:1}==6 ]]; then
        printf "This OS is running ${BLUE}$rel${DEFAULTC} which is compatible with this Bigtop release.\n\n"
else
        printf "This OS is ${RED}not compatible${DEFAULTC} with this Bigtop release. Please install CentOS 6.x\n\n"
        exit 1
fi


# Get hostname
printf "Enter the hostname of the master node or press Enter to use ${RED}$(hostname -f)${DEFAULTC} : "
read mnode
mnode=${mnode:-$(hostname -f)}
echo


# Install Dependencies
sudo yum -y install git

 
# Install Puppet
sudo rpm -ivh http://yum.puppetlabs.com/puppetlabs-release-el-6.noarch.rpm
sudo yum -y install puppet
sudo puppet module install puppetlabs-stdlib

 
# Install Bigtop Puppet
sudo git clone https://github.com/apache/bigtop.git /bigtop-home
sudo sh -c "cd /bigtop-home; git checkout branch-1.2"
sudo cp -r /bigtop-home/bigtop-deploy/puppet/hieradata/ /etc/puppet/
sudo cp /bigtop-home/bigtop-deploy/puppet/hiera.yaml /etc/puppet/


# ConfigurationConfigure
sudo su root -c "cat > /etc/puppet/hieradata/site.yaml <<- EOF
        ---
        bigtop::hadoop_head_node: "bigtop1.apache.org$mnode"
        hadoop::hadoop_storage_dirs:
        - /data/1
        - /data/2
        hadoop_cluster_node::cluster_components:
        - hdfs
        - spark
        bigtop::jdk_package_name: "java-1.8.0-openjdk-devel.x86_64"
        bigtop::bigtop_repo_uri: "http://bigtop-repos.s3.amazonaws.com/releases/1.2.0/centos/6/x86_64"
        EOF
"

 
# Deploy
sudo puppet apply --parser future --modulepath=/bigtop-home/bigtop-deploy/puppet/modules:/etc/puppet/modules /bigtop-home/bigtop-deploy/puppet/manifests




DONE!

Important settings

  • Customize to Hadoop+Spark+HBase Stack

    Code Block
    hadoop_cluster_node::cluster_components:
    - hdfs
    - yarn
    - spark
    - hbase
  • Bigtop 1.2.0 CentOS 6 repo: 

    Code Block
    bigtop::bigtop_repo_uri: "http://bigtop-repos.s3.amazonaws.com/releases/1.2.0/centos/6/x86_64"

    Check out available OS from Bigtop Installable binary artifacts