Tools needed
- JDK 1.6.0
- Apache Maven
- NodeJS and NPM
- git
Checking out Ambari source
The following uses Ambari git for the development process. Please keep in mind git is read only and patches will have to be uploaded and committed via svn.
Code Block |
---|
git clone git://git.apache.org/ambari.git
cd ambari
|
Follow the instructions under Checkout source code section of "How to contribute" guide.
We'll refer to the top-level "ambari" directory as AMBARI_DIR
in this document.
Building Ambari
After checking out Ambari, you can build ambari with the steps below:
Code Block |
---|
$ mvn clean install
|
Note: The above is for building the jars. See below for starting/deploying the Ambari Server and the Ambari Agent.
Building Artifacts and Running the Server/Agent
To build tarball artifacts run:
Code Block |
---|
$ mvn clean package assembly:assembly
|
This will build the Server and Agent tarballs.
NOTE: Run everything as root
below.
Running the Ambari Server:
Execute the following, where INSTALL_DIR
is a directory where you want to run the Server:
Code Block |
---|
$ cp ambari-server/target/ambari-server-*-dist.tar.gz INSTALL_DIR
$ cd INSTALL_DIR
$ tar -zxvf ambari-server-*-dist.tar.gz
|
Edit the Ambari properties configuration file:
Code Block |
---|
$ vim /etc/ambari/ambari.properties
|
Add the following entry:
Code Block |
---|
security.server.keys_dir=INSTALL_DIR/ambari-server-1.0.3-SNAPSHOT/keystore
|
NOTE: replace INSTALL_DIR
with the actual directory path.
Code Block |
---|
$ INSTALL_DIR
$ cd ambari-server-*
$ export AMBARI_PASSPHRASE="tmp"
$ cp /etc/ambari/ambari.properties etc/ambari-server/conf/
$ java -cp etc/ambari-server/conf/:lib/ambari-server/* org.apache.ambari.server.controller.AmbariServer
|
NOTE: the conf
directory should come first in the java classpath because guice-persist.jar
includes a log4j.properties
which will cause the Ambari log4j.properties
to not load.
The Server will launch and run as a foreground process. Scripting will come later as we move along to run the server with proper confs/envs.
To check if the server is running and healthy, you can go to:
Code Block |
---|
http://localhost:8080/api/check
|
The status will show as running. Or from a command line, you can run:
Code Block |
---|
curl localhost:8080/api/check
|
NOTE: The health check is static response for now.
Running the Ambari Agent.
The Ambari Agent requires python 2.6 to run on a host. RHEL5/CentOS5 has python 2.4 installed by default. To install python 2.6 (NOTE: you can install both 2.4/2.6 on the host), install the epel repo and then run:
Code Block |
---|
$ yum install python26.
|
Python needs setuptools to be able to setup the project for development. Download http://pypi.python.org/packages/2.6/s/setuptools/setuptools-0.6c11-py2.6.egg#md5=bfa92100bd772d5a213eedd356d64086 and run:
Code Block |
---|
$ sh setuptools-0.6c11-py2.6.egg
|
To setup the Ambari Agent:
Code Block |
---|
$ mkdir -p /var/ambari /var/run/ambari /var/log/ambari /tmp/ambari-agent/
|
The following dev process is easier than using setup.py and is a quicker turnaround. As soon as we have a good deployment story (rpm) we will add those in.
Code Block |
---|
$ cd ambari-agent/src/main/python/ambari_agent
|
- Ambari agent requires
facter
to get the node host information. Downloadfacter
tarball from http://downloads.puppetlabs.com/facter/facter-1.6.10.tar.gz and uninstall in a directory. - Edit facter_home in AmbariConfig.py to the directory where facter is installed. The agent executes the script facter_home/bin/facter to get all the node information and also uses df to get the mount point information.
- Ambari agent requires
puppet
. Downloadpuppet
tarball from http://www.puppetlabs.com/downloads/puppet/puppet-2.7.9.tar.gz and uninstall in a directory. - Edit puppet_home in AmbariConfig.py to the directory where puppet is installed. The agent executes puppet_home/bin/puppet to run puppet scripts.
To run the agent:
Code Block |
---|
$ export AMBARI_PASSPHRASE="tmp"
|
Copy puppet modules to /var/lib/ambari-agent/puppet/
Code Block |
---|
$ cp -R ../../puppet/modules/ /var/lib/ambari-agent/puppet
$ python2.6 main.py
|
The Ambari Agent will just try connecting to the server. To see the logs, take a look at:
Code Block |
---|
/var/log/ambari-agent/ambari-agent.log
|
Building and Running Ambari Web
Ambari Web uses Brunch (http://brunch.io/) as the application assembler for development purposes. Brunch is used to build and deploy automatically in the background as the developer modifies source files (JS, LESS/CSS, Handlebars template files). Brunch can also start up a Node.js-based web server with a single command so that you can easily run the application from the application root directory without any special configuration.
To build, deploy and run Ambari Web:
- Install NodeJS and NPM from http://nodejs.org/download/
- Install Brunch
Code Block $ npm install -g brunch
- Download dependent packages into ambari-web/node_modules directory
Code Block $ cd AMBARI_DIR/ambari-web $ npm install
To run Ambari Web so that it works with mock data independently from Ambari Server:
- Start test server:
Code Block $ brunch watch --server (or simply "brunch w --s")
- Edit
ambari-web/app/initialize.js
and set to test mode (testMode = true
). You can optionally set to go to the install wizard or directly into the app dashboard.Code Block App.testMode = true; App.skipBootstrap = true; App.alwaysGoToInstaller = true;
- Access http://localhost:3333 from your web browser. When you change Ambari Web source files, changes are immediately reflected.
To run Ambari Web with Ambari Server integration:
Code Block |
---|
$ cd AMBARI_DIR
$ mvn clean package assembly:assembly
|
Note: currently the mvn command does not put ambari-web in the Maven-generated package. This will be fixed in the future.
Code Block |
---|
$ cd AMBARI_DIR/ambari-server/ambari/target/ambari-server-*-dist/ambari-server-*
$ java -cp etc/ambari-server/conf/:lib/ambari-server/* org.apache.ambari.server.controller.AmbariServer
$ cd AMBARI_DIR/ambari-web
$ brunch watch
|
Access http://localhost:8080 from your web browser. When you change Ambari Web source files, changes are immediately reflected.
Setting up Ambari in Eclipse
Tools needed to build Ambari
The following tools are needed to build Ambari from source.
Alternatively, you can easily launch a VM that is preconfigured with all the tools that you need. See the Pre-Configured Development Environment section in the Quick Start Guide.
- xCode (if using Mac - free download from the apple store)
- JDK 8 (Ambari 2.6 and below can be compiled with JDK 7, from Ambari 2.7, it can be compiled with at least JDK 8)
- Apache Maven 3.3.9 or later
Tip: In order to persist your changes to the JAVA_HOME environment variable and add Maven to your path, create the following files:File: ~/.profileCode Block source ~/.bashrc
File: ~/.bashrcCode Block export PATH=/usr/local/apache-maven-3.3.9/bin:$PATH export JAVA_HOME=$(/usr/libexec/java_home) export _JAVA_OPTIONS="-Xmx2048m -XX:MaxPermSize=512m -Djava.awt.headless=true"
- Python 2.6 (Ambari 2.7 or later require Python 2.7 as minimum supported version)
Python setuptools:
for Python 2.6: Download setuptools and run:Code Block sh setuptools-0.6c11-py2.6.egg
for Python 2.7: Download setuptools and run:
Code Block sh setuptools-0.6c11-py2.7.egg
- rpmbuild (rpm-build package)
g++ (gcc-c++ package)
Running Unit Tests
mvn clean test
Run unit tests in a single module:
Code Block mvn -pl ambari-server test
Run only Java tests:
Code Block mvn -pl ambari-server -DskipPythonTests
Run only specific Java tests:
Code Block mvn -pl ambari-server -DskipPythonTests -Dtest=AgentHostInfoTest test
Run only Python tests:
Code Block mvn -pl ambari-server -DskipSurefireTests test
Run only specific Python tests:
Code Block mvn -pl ambari-server -DskipSurefireTests -Dpython.test.mask=TestUtils.py test
Run only Checkstyle and RAT checks:
Code Block mvn -pl ambari-server -DskipTests test
NOTE: Please make sure you have npm in the path before running the unit tests.
Generating Findbugs Report
mvn clean install
This will generate xml and html report unders target/findbugs. You can also add flags to skip unit tests to generate report faster.
Building Ambari
Note: if you can an error that too many files are open while building, then run: ulimit -n 10000 (for example)
To build Ambari RPMs, run the following.
Note: Replace ${AMBARI_VERSION} with a 4-digit version you want the artifacts to be (e.g., -DnewVersion=1.6.1.1)
Note: If running into errors while compiling the ambari-metrics package due to missing the artifacts of jms, jmxri, jmxtools:
[ERROR] Failed to execute goal on project ambari-metrics-kafka-sink: Could not resolve dependencies for project org.apache.ambari:ambari-metrics-kafka-sink:jar: 2.0 . 0 - 0 : The following artifacts could not be resolved: javax.jms:jms:jar: 1.1 , com.sun.jdmk:jmxtools:jar: 1.2 . 1 , com.sun.jmx:jmxri:jar: 1.2 . 1 : Could not transfer artifact javax.jms:jms:jar: 1.1 from/to java.net (https: //maven-repository.dev.java.net/nonav/repository): No connector available to access repository java.net (https://maven-repository.dev.java.net/nonav/repository) of type legacy using the available factories WagonRepositoryConnectorFactory |
The work around is to manually install the three missing artifacts:
mvn install:install-file -Dfile=jms- 1.1 .pom -DgroupId=javax.jms -DartifactId=jms -Dversion= 1.1 -Dpackaging=jar mvn install:install-file -Dfile=jmxtools- 1.2 . 1 .pom -DgroupId=com.sun.jdmk -DartifactId=jmxtools -Dversion= 1.2 . 1 -Dpackaging=jar mvn install:install-file -Dfile=jmxri- 1.2 . 1 .pom -DgroupId=com.sun.jmx -DartifactId=jmxri -Dversion= 1.2 . 1 -Dpackaging=jar |
If when compiling it seems stuck, and you've already increased Java and Maven heapsize, it could be that Ambari Views has a lot of artifacts, and the rat-check is choking up. In this case, try running
git clean -df (this will remove untracked files and directories) mvn clean package -DskipTests -Drat.ignoreErrors=true or mvn clean package -DskipTests -Drat.skip |
Setting the Version Using Maven
Ambari 2.8+ uses a newer method to update the version when building Ambari.
Code Block | ||||
---|---|---|---|---|
| ||||
# Update the revision property to the release version
mvn versions:set-property -Dproperty=revision -DnewVersion=2.8.0.0.0
mvn -B clean install package rpm:rpm -DskipTests -Dpython.ver="python >= 2.6" -Preplaceurl |
Code Block | ||||
---|---|---|---|---|
| ||||
# Update the revision property to the release version
mvn versions:set-property -Dproperty=revision -DnewVersion=2.8.0.0.0
mvn -B clean install package rpm:rpm -DskipTests -Psuse11 -Dpython.ver="python >= 2.6" -Preplaceurl |
Code Block | ||||
---|---|---|---|---|
| ||||
# Update the revision property to the release version
mvn versions:set-property -Dproperty=revision -DnewVersion=2.8.0.0.0
mvn -B clean install package jdeb:jdeb -DskipTests -Dpython.ver="python >= 2.6" -Preplaceurl |
Ambari 2.7 and Earlier Releases (Deprecated)
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
mvn versions:set -DnewVersion=${AMBARI_VERSION}
#Note: The ambari-metrics project is not wired up to the main ambari project. However there is a dependency on ambari-metrics-common to build the ambari-server RPM.
#Hence you also need to set ambari-metrics project version as well.
pushd ambari-metrics
mvn versions:set -DnewVersion=${AMBARI_VERSION}
popd
mvn -B clean install package rpm:rpm -DskipTests -Dpython.ver="python >= 2.6" -Preplaceurl
|
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
mvn versions:set -DnewVersion=${AMBARI_VERSION}
#Note: The ambari-metrics project is not wired up to the main ambari project. However there is a dependency on ambari-metrics-common to build the ambari-server RPM.
#Hence you also need to set ambari-metrics project version as well.
pushd ambari-metrics
mvn versions:set -DnewVersion=${AMBARI_VERSION}
popd
mvn -B clean install package rpm:rpm -DskipTests -Psuse11 -Dpython.ver="python >= 2.6" -Preplaceurl |
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
mvn versions:set -DnewVersion=${AMBARI_VERSION}
#Note: The ambari-metrics project is not wired up to the main ambari project. However there is a dependency on ambari-metrics-common to build the ambari-server RPM.
#Hence you also need to set ambari-metrics project version as well.
pushd ambari-metrics
mvn versions:set -DnewVersion=${AMBARI_VERSION}
popd
mvn -B clean install package jdeb:jdeb -DskipTests -Dpython.ver="python >= 2.6" -Preplaceurl |
Ambari Server will create following packages
- RPM will be created under
AMBARI_DIR
/ambari-server/target/rpm/ambari-server/RPMS/noarch. - DEB will be created under
AMBARI_DIR
/ambari-server/target/
Ambari Agent will create following packages
- RPM will be created under
AMBARI_DIR
/ambari-agent/target/rpm/ambari-agent/RPMS/x86_64. - DEB will be created under
AMBARI_DIR
/ambari-agent/target
Optional parameters:
- -X -e: add these options for more verbose output by Maven. Useful when debugging Maven issues.
- -DdefaultStackVersion=STACK-VERSION
- Sets the default stack and version to be used for installation (e.g., -DdefaultStackVersion=HDP-1.3.0)
- -DenableExperimental=true
- Enables experimental features to be available via Ambari Web (default is false)
- All views can be packaged in RPM by adding -Dviews parameter
- mvn -B clean install package rpm:rpm -Dviews -DskipTests
- Specific views can be built by adding
--projects
parameter to the -Dviews- mvn -B clean install package rpm:rpm --projects ambari-web,ambari-project,ambari-views,ambari-admin,contrib/views/files,contrib/views/pig,ambari-server,ambari-agent,ambari-client,ambari-shell -Dviews -DskipTests
- mvn -B clean install package rpm:rpm --projects ambari-web,ambari-project,ambari-views,ambari-admin,contrib/views/files,contrib/views/pig,ambari-server,ambari-agent,ambari-client,ambari-shell -Dviews -DskipTests
NOTE: Run everything as root
below.
Building Ambari Metrics
If you plan on installing the Ambari Metrics service, you will also need to build the Ambari Metrics project.
Code Block |
---|
cd ambari-metrics
mvn clean package -Dbuild-rpm -DskipTests
For Ubuntu:
cd ambari-metrics
mvn clean package -Dbuild-deb -DskipTests |
Note:
The metrics rpms will be found at: ambari-metrics-assembly/target/. These would be need for installing the Ambari Metrics service.
Running the Ambari Server
First, install the Ambari Server RPM.
On RHEL/CentOS:
Code Block |
---|
yum install ambari-server/target/rpm/ambari-server/RPMS/noarch/ambari-server-*.noarch.rpm
|
On SUSE/SLES:
Code Block |
---|
zypper install ambari-server/target/rpm/ambari-server/RPMS/noarch/ambari-server-*.noarch.rpm
|
On Ubuntu 12:
Code Block |
---|
dpkg --install ambari-server/target/ambari-server-*.deb # Will fail with missing dependencies errors
apt-get update # Update locations of dependencies
apt-get install -f # Install all failed dependencies
dpkg --install ambari-server/target/ambari-server-*.deb # Will succeed |
Initialize Ambari Server:
Code Block |
---|
ambari-server setup
|
Start up Ambari Server:
Code Block |
---|
ambari-server start
|
See Ambari Server log:
Code Block |
---|
tail -f /var/log/ambari-server/ambari-server.log
|
To access Ambari, go to
Code Block |
---|
http://{ambari-server-hostname}:8080 |
from your web browser and log in with username admin and password admin.
Install and Start the Ambari Agent Manually on Each Host in the Cluster
Install the Ambari Agent RPM.
On RHEL/CentOS:
Code Block |
---|
yum install ambari-agent/target/rpm/ambari-agent/RPMS/x86_64/ambari-agent-*.rpm
|
SUSE/SLES:
Code Block |
---|
zypper install ambari-agent/target/rpm/ambari-agent/RPMS/x86_64/ambari-agent-*.rpm
|
Ubuntu12:
Code Block |
---|
dpkg --install ambari-agent/target/ambari-agent-*.deb |
Edit the location of Ambari Server in /etc/ambari-agent/conf/ambari-agent.ini by editing the hostname line.
Start Ambari Agent:
Code Block |
---|
ambari-agent start
|
See Ambari Agent log:
Code Block |
---|
tail -f /var/log/ambari-agent/ambari-agent.log
|
Setting up Ambari in Eclipse
Code Block |
---|
Code Block |
$ mvn clean eclipse:eclipse
|
After doing the above you should be able to import the project via Eclipse "Import > Maven > Existing Maven Project". Choose the root directory where you cloned the git repository. You should be able to see the following projects on eclipse:
Code Block |
---|
ambari
|
|- ambari-project
|- ambari-server
|- ambari-agent
|- ambari-web
|
Select the top-level "ambari pom.xml" and click Finish.