There are three main steps for installing Ambari and HDP using VirtualBox and Vagrant:
- Install VirtualBox and Vagrant.
VirtualBox allows you to run multiple operating systems (inside multiple virtual machines) at the same time. In this Quick Start you will be using VirtualBox to run Linux on OS X.
- Vagrant is a tool that makes it easier to work with virtual machines. It helps automate the work of setting up, running, and removing virtual machine environments. For our purposes, Vagrant makes it easy to install and run a preconfigured environment for running Ambari.
- Start one or more VMs, each representing a node in a cluster.
- Download, install, and deploy the version of Ambari you wish to use.
- From Ambari, deploy the version of HDP you wish to use.
Once VirtualBox and Vagrant have been installed, steps 2 through 4 can be done multiple times (to change versions, create a larger cluster, etc.).
Note: these steps have been tested on MacOS 10.9.5.
Installing VirtualBox and Vagrant
- Download and install VirtualBox from https://www.virtualbox.org/wiki/Downloads. Note: as of 4/25/16 there are potential issues running the latest version of VirtualBox with the Ambari Vagrant setup. We recommend installing an older (4.x) version. This Quick Start has been tested on 4.3.34.
- Download and install Vagrant from http://downloads.vagrantup.com.
Clone the
ambari-vagrant
GitHub repository into a convenient folder on your Mac. The repository contains scripts for setting up Ambari virtual machines on several Linux distributions.
From the command-line:git clone https://github.com/u39kun/ambari-vagrant.git
Edit the
/etc/hosts
file on your computer so that you will be able to resolve hostnames for the VMs:sudo -s 'cat ambari-vagrant/append-to-etc-hosts.txt >> /etc/hosts'
Use the
vagrant
command to create a private key. You will need the key for Ambari.
Thevagrant
command displays Vagrant command usage and common commands, and creates a private key as~/.vagrant.d/insecure_private_key
.vagrant
Start the Linux Virtual Machines
The ambari-vagrant
directory (cloned from GitHub) contains several subdirectories, each for a specific Linux distribution. Each subdirectory has scripts and configuration files for running Ambari and HDP on that version of Linux.
To start one or more virtual machines:
Change your current directory to
ambari-vagrant
:cd ambari-vagrant
You will see subdirectories for different OS’s.
cd
into the OS subdirectory for the OS you wish to use. CentOS 6.4 is recommended, because it is quicker to launch than other OS's.
The remainder of this Quick Start uses CentOS 6.4. To install and use another version or distribution of Linux, specify the other directory name in place of centos6.4cd centos6.4
Copy the private key into your OS directory:
cp ~/.vagrant.d/insecure_private_key .
(Optional) If you have at least 16 GM of memory on your Mac, consider increasing the amount of memory allocated to the VMs. Edit the following line in
Vagrantfile
, increasing 3072 to 4096 or more:vb.customize ["modifyvm", :id, "--memory", 3072] # RAM allocated to each VM
Start one or more VMs, using the
./up.sh
command. Three seems to be a good number if you have 16GB of RAM, without taxing the system too much../up.sh <# of VMs to launch>
For example,
up.sh 3
on CentOS 6.4 starts 3 VMs. Associated hostnames will bec6401
,c6402
, andc6403
.Additional notes:
- With the defaultVagrantfile
(in each OS subdirectory) you can specify up to 10 VMs. (If your computer can handle it; you can even add more.)
- VMs will have fully-qualified domain names (FQDN) with the format<os-code>[01-10].ambari.apache.org
, where<os-code>
isc59
(CentOS 5.9),c64
(CentOS 6.4), etc. For example,c5901.ambari.apache.org
,c6401.ambari.apache.org
, etc.
- VMs will have IP addresses192.168.<os-subnet>.1[01-10]
, where<os-subnet>
is59
for CentOS 5.9,64
for CentOS 6.4, etc. For example,192.168.59.101
for CentOS 5.9 nodec5901
,192.168.64.101
, for CentOS 6.4 nodec6401
, etc.
- Note thatup.sh 3
is equivalent to doing something likevagrant up c640{1..3}
Check the status of your VMs:
hw100:centos6.4 lkg$ vagrant status Current machine states: c6401 running (virtualbox) c6402 running (virtualbox) c6403 running (virtualbox) c6404 not created (virtualbox) c6405 not created (virtualbox) c6406 not created (virtualbox) c6407 not created (virtualbox) c6408 not created (virtualbox) c6409 not created (virtualbox) c6410 not created (virtualbox)
(You can use the
vagrant init
command to initialize your current directory as a Vagrant environment, but this step should already be handled by the preceding steps.)Next, log into one of your VMs. For example:
vagrant ssh c6401
From this point onward, your terminal window will operate within the VM until you exit the VM. All commands will go to the VM, not your Mac.
When you first access the VM you will be logged in as user
vagrant
. Change toroot
user:sudo su -
Install Ambari
Prerequisites: The following software must be installed on your VM:
yum
rpm
scp
curl
wget
pdsh
On your VM (where the main ambari service should run?), download the Ambari repository. The following commands download Ambari version 2.2.1.0 and install
ambari-server
:OS Commands CentOS 6 wget -O /etc/yum.repos.d/ambari.repo http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.2.1.0/ambari.repo yum install ambari-server -y
CentOS 7 wget -O /etc/yum.repos.d/ambari.repo http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.2.1.0/ambari.repo
yum install ambari-server -yUbuntu 12 wget -O /etc/apt/sources.list.d/ambari.list http://public-repo-1.hortonworks.com/ambari/ubuntu12/2.x/updates/2.2.1.0/ambari.list
apt-key adv --recv-keys --keyserver keyserver.ubuntu.com B9733A7A07513CAD apt-get update apt-get install ambari-server -yUbuntu 14 wget -O /etc/apt/sources.list.d/ambari.list http://public-repo-1.hortonworks.com/ambari/ubuntu14/2.x/updates/2.2.1.0/ambari.list
apt-key adv --recv-keys --keyserver keyserver.ubuntu.com B9733A7A07513CAD
apt-get update
apt-get install ambari-server -yNote: to test a branch build or the trunk build, specify the repo URL for the branch or trunk version.
Ambari offers many installation options (see Ambari User Guides), but to install Ambari with default settings, run the following commands to set up and start
ambari-server
:ambari-server setup -s ambari-server start
For frontend developers only: see Frontend Development section below for extra setup instructions.
Once Ambari Server is started, hit http://c6401.ambari.apache.org:8080 (URL depends on the OS being tested) from your browser on your local computer.
Note that Ambari Server can take some time to fully come up and ready to accept connections. Keep hitting the URL until you get the login page.
Once you are at the login page, login with the default username admin and password admin.
On the Install Options page, use the FQDNs of the VMs. For example:
c6401.ambari.apache.org c6402.ambari.apache.org c6403.ambari.apache.org
Alternatively, you can use a range expression:
c64[01-03].ambari.apache.org
Specify the the non-root SSH user vagrant, and upload insecure_private_key file that you copied earlier as the private key.
Follow the onscreen instructions to install your cluster.
When done testing, run vagrant destroy -f to purge the VMs.
Basic VM Operations
vagrant up <vm name>
Starts a specific VM. up.sh is a wrapper for this call.
Note: if you don’t specify the <vm name> parameter, it will try to start 10 VMs
You can run this if you want to start more VMs after you already called up.sh
For example: vagrant up c6406
vagrant destroy -f
Destroys all VMs launched from the current directory (deletes them from disk as well)
You can optionally specify a specific VM to destroy
vagrant suspend
Suspends (snapshot) all VMs launched from the current directory so that you can resume them later
You can optionally specify a specific VM to suspend
vagrant resume
Resumes all suspended VMs launched from the current directory
You can optionally specify a specific VM to resume
vagrant ssh host
Starts a SSH session to the host. For example: vagrant ssh c6401
vagrant status
Shows which VMs are running, suspended, etc.
Modifying RAM for the VMs
Each VM is allocated 2GB of RAM. These can be changed by editing Vagrantfile. To change the RAM allocation, modify the following line:
vb.customize ["modifyvm", :id, "--memory", 2048]
Taking Snapshots
Vagrant makes it easy to take snapshots of the entire cluster.
First, install the snapshot plugin:
vagrant plugin install vagrant-vbox-snapshot --plugin-version=0.0.2
This enables the “vagrant snapshot” command. Note that the above installs vesion 0.0.2. if you install the latest plugin version 0.0.3 does not allow taking snapshots of the whole cluster at the same time (you have to specify a VM name).
Run vagrant snapshot to see the syntax.
Note that the plugin tries to take a snapshot of all VMs configured in Vagrantfile. If you are always using 3 VMs, for example, you can comment out c64[04-10] in Vagrantfile so that the snapshot commands only operate on c64[01-03].
Note: Upon resuming a snapshot, you may find that time-sensitive services may be down (e.g, HBase RegionServer is down, etc.)
Tip: After starting the VMs but before you do anything on the VMs, run "vagrant snapshot take init". This way, you can go back to the initial state of the VMs by running "vagrant snapshot go init"; this only takes seconds (much faster than starting the VMs up from scratch by using up.sh or "vagrant up"). Another advantage of this is that you can always go back to the initial state without destroying other named snapshots that you created.
Misc
All VMs launched will have a directory called /vagrant inside the VM. This maps to the ambari-vagrant/<os> directory on your local computer. You can use this shared directory mapping to push files, etc.
If you want to test OS’s other than what’s currently in the ambari-vagrant repo, please see http://www.vagrantbox.es/ for all the readily available OS images you can test. Note that Ambari currently works on RHEL 5/6, CentOS 5/6, Oracle Linux 5/6, SUSE 11, and SLES 11. Ubuntu support is work in progress.
Pre-Configured Development Environment
Simply edit Vagrantfile to launch a VM with all the tools necessary to build Ambari from source.
cd ambari-vagrant/centos6.4 vi Vagrantfile <- uncomment the line with "dev-bootstrap.sh" vagrant up c6401
To build from source, follow the instructions in Ambari Development.
Frontend Development
You can use this set up to develop and test out Ambari Web frontend code against a real Ambari Server on a multi-node environment.
On the host machine:
cd ambari-vagrant/centos6.4 git clone https://git-wip-us.apache.org/repos/asf/ambari.git ambari cd ambari/ambari-web npm install brunch w
On c6401 (where Ambari Server is installed):
cd /usr/lib/ambari-server mv web web-orig ln -s /vagrant/ambari/ambari-web/public web ambari-server restart
With this setup, whenever you change the content of ambari-web files (under ambari-vagrant/ambari/) on the host machine, brunch will pick up changes in the background and update ambari-vagrant/ambari/ambari-web/public. Because of the symbolic link, the changes are automatically picked up by Ambari Server. All you have to do is hit refresh on the browser to see the frontend code changes reflected.
Not seeing code changes as expected? If you have run the maven command to build Ambari previously, you will see files called app.js.gz and vendor.js.gz under the public folder. You need to delete these files for the frontend code changes to be effective, as the app.js.gz and vendor.js.gz files take precedence over app.js and vendor.js, respectively.