...
- Install VirtualBox and Vagrant.
VirtualBox allows you to run multiple operating systems (inside multiple virtual machines) at the same time. In this Quick Start you will be using VirtualBox to run Linux on OS X.
- Vagrant is a tool that makes it easier to work with virtual machines. It helps automate the work of setting up, running, and removing virtual machine environments. For our purposes, Vagrant makes it easy to install and run a preconfigured environment for running Ambari.
- Start one or more VMs, each representing a node in a cluster.
- Download, install, and deploy the version of Ambari you wish to use.
- From Ambari, deploy the version of HDP you wish to use.
- Start one or more VMs, each representing a node in a cluster.
- Download, install, and deploy the version of Ambari you wish to use.
- From Ambari, deploy the version of HDP you wish to use.
Once Once VirtualBox and Vagrant have been installed, steps 2 through 4 can be done multiple times (to change versions, create a larger cluster, etc.).
...
Installing VirtualBox and Vagrant
VirtualBox allows you to run multiple operating systems (inside multiple virtual machines) at the same time. In this Quick Start you will be using VirtualBox to run Linux on OS X.
Vagrant is a tool that makes it easier to work with virtual machines. It helps automate the work of setting up, running, and removing virtual machine environments. For our purposes, Vagrant makes it easy to install and run a preconfigured environment for running Ambari.
- Download and install VirtualBox from https:Download and install VirtualBox from https://www.virtualbox.org/wiki/Downloads.
Note: as of 4/25/16 there are potential issues running the latest version of VirtualBox with the Ambari Vagrant setup. We recommend installing an older (4.x) version. This Quick Start has been tested on 4.3.34. - Download and install Vagrant from http://downloads.vagrantup.com.
Clone the
ambari-vagrant
GitHub repository into a convenient folder on your Mac. The repository contains scripts for setting up Ambari virtual machines on several Linux distributions.
From the command-line:Code Block git clone https://github.com/u39kun/ambari-vagrant.git
Edit the
/etc/hosts
file on your computer so that you will be able to resolve hostnames for the VMs:Code Block sudo -s 'cat ambari-vagrant/append-to-etc-hosts.txt >> /etc/hosts'
Use the
vagrant
command to create a private key. You will need the key for Ambari.
Thevagrant
command displays Vagrant command usage and common commands, and creates a private key as~/.vagrant.d/insecure_private_key
.Code Block vagrant
...
Change your current directory to
ambari-vagrant
:Code Block cd ambari-vagrant
You will see subdirectories for different OS’s.
cd
into the OS subdirectory for the OS you wish to use. CentOS 6.4 is recommended, because it is quicker to launch than other OS's.
The remainder of this Quick Start uses CentOS 6.4. To install and use another version or distribution of Linux, specify the other directory name in place of centos6.4Code Block cd centos6.4
Copy the private key into your OS directory:
Code Block cp ~/.vagrant.d/insecure_private_key .
(Optional) If you have at least 16 GB of memory on your Mac, consider increasing the amount of memory allocated to the VMs. Edit the following line in
Vagrantfile
, increasing 3072 to 4096 or more:Code Block vb.customize ["modifyvm", :id, "--memory", 3072] # RAM allocated to each VM
- All VMs will have a directory called
/vagrant
inside the VM. This corresponds to theambari-vagrant/<os>
directory on your local computer. If you have any files to access from within the VM, you can place them in this shared directory. Start one or more VMs, using the
./up.sh
command. Three seems to be a good number if you have 16GB of RAM, without taxing the system too much.Code Block ./up.sh <# of VMs to launch>
For example,
up.sh 3
on CentOS 6.4 starts 3 VMs. Associated hostnames will bec6401
,c6402
, andc6403
.Additional notes:
- With the defaultVagrantfile
(in each OS subdirectory) you can specify up to 10 VMs. (If your computer can handle it; you can even add more.)
- VMs will have fully-qualified domain names (FQDN) with the format<os-code>[01-10].ambari.apache.org
, where<os-code>
isc59
(CentOS 5.9),c64
(CentOS 6.4), etc. For example,c5901.ambari.apache.org
,c6401.ambari.apache.org
, etc.
- VMs will have IP addresses192.168.<os-subnet>.1[01-10]
, where<os-subnet>
is59
for CentOS 5.9,64
for CentOS 6.4, etc. For example,192.168.59.101
for CentOS 5.9 nodec5901
,192.168.64.101
, for CentOS 6.4 nodec6401
, etc.
- Note thatup.sh 3
is equivalent to doing something likevagrant up c640{1..3}
Check the status of your VMs:
Code Block hw100:centos6.4 lkg$ vagrant status Current machine states: c6401 running (virtualbox) c6402 running (virtualbox) c6403 running (virtualbox) c6404 not created (virtualbox) c6405 not created (virtualbox) c6406 not created (virtualbox) c6407 not created (virtualbox) c6408 not created (virtualbox) c6409 not created (virtualbox) c6410 not created (virtualbox)
(You can use the
vagrant init
command to initialize your current directory as a Vagrant environment, but this step should already be handled by the preceding steps.)Next, log into one of your VMs. For example:
Code Block vagrant ssh c6401
From this point onward, this terminal window will operate within the VM until you exit the VM. All commands will go to the VM, not your Mac.
Tip: Open a second terminal window for submitting commands on your Mac. This will be useful when accessing the Ambari Web UI.When you first access the VM you will be logged in as user
vagrant
. Change toroot
user:Code Block sudo su - -
- When you are finished using the VM, TBD (save, exit; snapshot)
Install Ambari
Prerequisites: Before installing Ambari, the following software must be installed on your VM:
yum
rpm
scp
curl
wget
pdsh
- ntpd?
(how to check)
On From the terminal window on your VM (where the main ambari service should run?), download the Ambari repository. The following commands download Ambari version 2.2.1.0 and install
ambari-server
:OS Commands CentOS 6 wget -O /etc/yum.repos.d/ambari.repo http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.2.1.0/ambari.repo yum install ambari-server -y
CentOS 7 wget -O /etc/yum.repos.d/ambari.repo http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.2.1.0/ambari.repo
yum install ambari-server -yUbuntu 12 wget -O /etc/apt/sources.list.d/ambari.list http://public-repo-1.hortonworks.com/ambari/ubuntu12/2.x/updates/2.2.1.0/ambari.list
apt-key adv --recv-keys --keyserver keyserver.ubuntu.com B9733A7A07513CAD apt-get update apt-get install ambari-server -yUbuntu 14 wget -O /etc/apt/sources.list.d/ambari.list http://public-repo-1.hortonworks.com/ambari/ubuntu14/2.x/updates/2.2.1.0/ambari.list
apt-key adv --recv-keys --keyserver keyserver.ubuntu.com B9733A7A07513CAD
apt-get update
apt-get install ambari-server -yNote: to test a branch build or the trunk build, specify the repo URL for the branch or trunk version.
To install Ambari with default settings, run the following commands to set up and start
ambari-server
:Code Block ambari-server setup -s ambari-server start
For more information about installation options, see Ambari User Guides.
(Optional) For frontend developers only: see "Frontend Development" (later on this page) for extra setup instructions.
Once Ambari Server is started, access the Ambari Web UI at
http://<hostname>.ambari.apache.org:8080
. For example:Code Block http://c6401.ambari.apache.org:8080
The URL depends on the OS and the node where you installed Ambari.
Note: The Ambari Server can take some time to launch and be ready to accept connections. Keep trying the URL until you see the login page.Login using default username
admin
, passwordadmin
.Choose "Launch Install Wizard."
Name your cluster.
On the Install Options page, list the FQDNs of the VMs. For examplethe VMs. For example:
Code Block c6401.ambari.apache.org c6402.ambari.apache.org c6403.ambari.apache.org
Alternatively, you can use a range expression:
Code Block c6401c64[01-03].ambari.apache.org c6402.ambari.apache.org c6403.ambari.apache.org
Alternatively, use a range expression:
Code Block c64[01-03].ambari.apache.org
Upload the insecure_private_key file that you created earlier.
Specify non-root SSH user
vagrant
.Follow the onscreen instructions to install your cluster.
Install the HDP Stack
When done testing, run
Upload the
insecure_private_key
file that you created earlier.Specify non-root SSH user
vagrant
.Follow the onscreen instructions to install your cluster.
Install the HDP Stack
<tbd>
Choose which HDP version, ...
Choose services
Assign master and slave processes
Customize services (here I assigned admin/admin to Hive and Oozie)
Timeline for my Mac & VMs to install
Other tips:
- yum failed - ?
- fix transparent huge page setting in /etc/rc.loca (CentOS7)
- "host checks" & message about firewall & THP: not very clear. Sounded like those were the warnings.
- ntpd not installed?
Basic Vagrant Commands
Command | Description |
---|---|
vagrant up <vm-name> | Starts a specific VM. ( Example: Note: if you do not specify the |
vagrant status [<vm-name>] | Shows which VMs are running, suspended, etc. |
vagrant destroy -f \[<vm-name>] | Destroys all VMs launched from the current directory, and deletes them from disk. You can optionally specify Optional: Specify a specific VM to destroy. |
vagrant suspend [<vm-name>] | Suspends (snapshot) all VMs launched from the current directory so that you can resume them later You can optionally specify Optional: Specify a specific VM to suspend. |
vagrant resume [<vm-name>] | Resumes all suspended VMs launched from the current directory You can optionally specify Optional: Specify a specific VM to resume. |
vagrant ssh host <vm-name> | Starts a SSH session to the host. For exampleExample: |
vagrant status | Shows which VMs are running, suspended, etc. |
--help | List information about Vagrant commands. |
Taking Snapshots
A Vagrant snapshot saves the current state of a VM, so that you can re-run the VM from the same point at a future time.
Vagrant makes it easy to take snapshots of the entire cluster.
First, install the snapshot plugin:
...
This enables the “vagrant snapshot” command. Note that the above above installs vesion 0.0.2. if you install the latest plugin version 0.0.3 does not not allow taking snapshots of the whole cluster at the same time (you have to specify a VM name).
Run vagrant snapshot to see the syntax.
Note that the plugin tries to take a snapshot of all VMs configured in Vagrantfile. If you are always using 3 VMs, for example, you can comment out c64[04-10] in Vagrantfile so that the snapshot commands only operate on c64[01-03].
Note: Upon resuming a snapshot, you may find that time-sensitive services may be down (e.g, HBase RegionServer is down, etc.)
Tip: After starting the VMs but before you do anything on the VMs, run "vagrant snapshot take init". This way, you can go back to the initial state of the VMs by running "vagrant snapshot go init"; this only takes seconds (much faster than starting the VMs up from scratch by using up.sh or "vagrant up"). Another advantage of this is that you can always go back to the initial state without destroying other named snapshots that you created.
Misc
If you want to To test OS’s other than what’s currently in the ambari-vagrant repo, please see http://www.vagrantbox.es/ for all the readily available OS images you can test. Note that Ambari currently works on RHEL 5/6, CentOS 5/6, Oracle Linux 5/6, SUSE 11, and SLES 11. Ubuntu support is work in progress.
...