Checking out Ambari source

The following uses the Ambari git repository for the development process. Please keep in mind that the git repository is read only and patches will have to be uploaded and committed via svn.

git clone git://git.apache.org/ambari.git
cd ambari

Follow the instructions under Checkout source code section of "How to contribute" guide.

We'll refer to the top-level "ambari" directory as AMBARI_DIR in this document.

Tools needed to build Ambari

The following tools are needed to build Ambari from source.

Alternatively, you can easily launch a VM that is preconfigured with all the tools that you need.  See the Pre-Configured Development Environment section in the Quick Start Guide.

• xCode (if using Mac - free download from the apple store)
• JDK 8 (Ambari 2.6 and below can be compiled with JDK 7, from Ambari 2.7, it can be compiled with at least JDK 8)
• Apache Maven 3.3.9 or later
  Tip: In order to persist your changes to the JAVA_HOME environment variable and add Maven to your path, create the following files:
  
  File: ~/.profile

   

  Code Block
  source ~/.bashrc

   

  File: ~/.bashrc

   

  Code Block
  export PATH=/usr/local/apache-maven-3.3.9/bin:$PATH export JAVA_HOME=$(/usr/libexec/java_home) export _JAVA_OPTIONS="-Xmx2048m -XX:MaxPermSize=512m -Djava.awt.headless=true"

  
  

• Python 2.6 (Ambari 2.7 or later require Python 2.7 as minimum supported version)

• Python setuptools:
  for Python 2.6: Download setuptools

• JDK 1.6.0
• Apache Maven
• Python 2.6
• Python setuptools - Download

  and run:

  Code Block
  sh setuptools-0.6c11-py2.6.egg

• NodeJS
• Brunch 1.4.4 (to install it, run the following command after NodeJS is installed):

  Code Block
  npm install -g brunch@1.4.4

Building Ambari

• for Python 2.7:  Download setuptools and run:

  Code Block
  sh setuptools-0.6c11-py2.7.egg

  
  

• rpmbuild (rpm-build package)

• g++ (gcc-c++ package)

Running Unit Tests

• mvn clean test

• Run unit tests in a single module:

  Code Block
  mvn -pl ambari-server test

  
  

• Run only Java tests:

  Code Block
  mvn -pl ambari-server -DskipPythonTests

  
  

• Run only specific Java tests:

  Code Block
  mvn -pl ambari-server -DskipPythonTests -Dtest=AgentHostInfoTest test

  
  

• Run only Python tests:

  Code Block
  mvn -pl ambari-server -DskipSurefireTests test

  
  

• Run only specific Python tests:

  Code Block
  mvn -pl ambari-server -DskipSurefireTests -Dpython.test.mask=TestUtils.py test

  
  

• Run only Checkstyle and RAT checks:

  Code Block
  mvn -pl ambari-server -DskipTests test

  
  

NOTE: Please make sure you have npm in the path before running the unit tests.

Generating Findbugs Report

• mvn clean install

This will generate xml and html report unders target/findbugs. You can also add flags to skip unit tests to generate report faster.

Building Ambari

Note: if you can an error that too many files are open while building, then run: ulimit -n 10000 (for example)

To build Ambari RPMs, run the following.

Note: Replace ${AMBARI_VERSION} with a 4-digit version you want the artifacts to be (e.g., -DnewVersion=1.6.1.1)

Note: If running into errors while compiling the ambari-metrics package due to missing the artifacts of jms, jmxri, jmxtools:

The work around is to manually install the three missing artifacts:

RHEL/CentOS 5:

# Update the revision property to the release version
mvn versions:set-property -Dproperty=revision -DnewVersion=2.8.0.0.0 

mvn -B clean install package rpm:rpm -DskipTests -Dpython.ver="python >= 2.6" -Preplaceurl

# Update the revision property to the release version
mvn versions:set-property -Dproperty=revision -DnewVersion=2.8.0.0.0 
 
mvn -B

...

 clean install package rpm:rpm -DskipTests -

...

Psuse11 -Dpython.ver="python >= 2.6" -Preplaceurl

# Update the revision property to the release version
mvn versions:set-property -Dproperty=revision -DnewVersion=2.8.0.0.0 
 
mvn -B clean install package jdeb:jdeb -DskipTests -Dpython.ver="python >= 2.6" -Preplaceurl

mvn versions:set -DnewVersion=${AMBARI_VERSION}
 
#Note: The ambari-metrics project is not wired up to the main ambari project. However there is a dependency on ambari-metrics-common to build the ambari-server RPM. 
#Hence you also need to set ambari-metrics project version as well.
pushd ambari-metrics
mvn versions:set -DnewVersion=${AMBARI_VERSION}
popd
 
mvn -B
mvn -X -B -e clean install package rpm:rpm -DskipTests -Dpython.ver="python >= 2.6" -Preplaceurl

...

mvn versions:set -DnewVersion=${AMBARI_VERSION}
 
#Note: The ambari-metrics project is not wired up to the main ambari project. However there is a dependency on ambari-metrics-common to build the ambari-server RPM. 
#Hence you also need to set ambari-metrics project version as well.
pushd ambari-metrics
mvn versions:set -DnewVersion=${AMBARI_VERSION}
popd
 
mvn -B
mvn -X -B -e clean install package rpm:rpm -DskipTests -Psuse11 -Dpython.ver="python >= 2.6" -Preplaceurl

mvn versions:set -DnewVersion=${AMBARI_VERSION}
 
#Note: The ambari-metrics project is not wired up to the main ambari project. However there is a dependency on ambari-metrics-common to build the ambari-server RPM. 
#Hence you also need to set ambari-metrics project version as well.
pushd ambari-metrics
mvn versions:set -DnewVersion=${AMBARI_VERSION}
popd
 
mvn -B clean install package jdeb:jdeb -DskipTests -Dpython.ver="python >= 2.6" -Preplaceurl

Ambari Server will create following packages

• RPM will be created under AMBARI_DIR/ambari-server/target/rpm/ambari-server/RPMS/noarch.
• DEB will be created under AMBARI_DIR/ambari-server/target/

Ambari Agent will create following packages

• RPM will be created under AMBARI_DIR/ambari-agent/target/rpm/ambari-agent/RPMS/x86_64.
• DEB will be created under AMBARI_DIR/ambari-agent/target

Optional parameters:

• -X -e: add these options for more verbose output by Maven.  Useful when debugging Maven issues.
• -DdefaultStackVersion=STACK-VERSION
• Sets the default stack and version to be used for installation (e.g., -DdefaultStackVersion=HDP-1.3.0)
• -DenableExperimental=true
• Enables experimental features to be available via Ambari Web (default is false)
• All views can be packaged in RPM by adding -Dviews parameter
  • mvn -B clean install package rpm:rpm -Dviews -DskipTests
• Specific views can be built by adding --projects parameter to the -Dviews
  • mvn -B clean install package rpm:rpm --projects ambari-web,ambari-project,ambari-views,ambari-admin,contrib/views/files,contrib/views/pig,ambari-server,ambari-agent,ambari-client,ambari-shell -Dviews -DskipTests
    
    

NOTE: Run everything as root below.

Building Ambari Metrics

If you plan on installing the Ambari Metrics service, you will also need to build the Ambari Metrics project. 

cd ambari-metrics
mvn clean package -Dbuild-rpm -DskipTests

For Ubuntu:
cd ambari-metrics
mvn clean package -Dbuild-deb -DskipTests

Note:

The metrics rpms will be found at: ambari-metrics-assembly/target/. These would be need for installing the Ambari Metrics service.

Running the Ambari Server

First, install the Ambari Server RPM.

On RHEL/CentOS:

yum install ambari-server/target/rpm/ambari-server/RPMS/noarch/ambari-server-*.noarch.rpm 

On SUSE/SLES:

zypper install ambari-server/target/rpm/ambari-server/RPMS/noarch/ambari-server-*.noarch.rpm 

On Ubuntu 12:

 
dpkg --install ambari-server/target/ambari-server-*.deb          # Will fail with missing dependencies errors
apt-get update                                                   # Update locations of dependencies
apt-get install -f                                               # Install all failed dependencies
dpkg --install ambari-server/target/ambari-server-*.deb          # Will succeed

Initialize Ambari Server:

ambari-server setup

Start up Ambari Server:

ambari-server start

See Ambari Server log:

tail -f /var/log/ambari-server/ambari-server.log

...

On RHEL/CentOS:

yum install ambari-agent/target/rpm/ambari-agent/RPMS/x86_64/ambari-agent-*.rpm

SUSE/SLES:

zypper install ambari-agent/target/rpm/ambari-agent/RPMS/x86_64/ambari-agent-*.rpm

Ubuntu12:

dpkg --install ambari-agent/target/ambari-agent-*.deb

Edit the location of Ambari Server in /etc/ambari-agent/conf/ambari-agent.ini by editing the hostname line.

Start Ambari Agent:

ambari-agent start

See Ambari Agent log:

tail -f /var/log/ambari-agent/ambari-agent.log

Setting up Ambari in Eclipse

$ mvn clean eclipse:eclipse

After doing the above you should be able to import the project via Eclipse "Import > Maven > Existing Maven Project". Choose the root directory where you cloned the git repository. You should be able to see the following projects on eclipse:

ambari
|
|- ambari-project
|- ambari-server
|- ambari-agent
|- ambari-web

Select the top-level "ambari pom.xml" and click Finish.

Coding Guidelines

...

Ambari Web Frontend Development Environment

Application Assembler: Brunch

We use Brunch as the client-side application assembler: http://brunch.io/^{Image Removed}
Brunch can:

Generate an application skeleton that is well structured with common client-side libraries (Backbone.js/Ember.js, jQuery, JavaScript/CoffeeScript, Stylus/LESS, etc)
Build and deploy automatically in the background as you modify the source files.
This lets you break up the application into a number of JS files for code organization and reuse without worrying about runtime load performance.
For QA/production deployment, we can turn on JavaScript/CSS minification for fast application loading.
Run automated tests.
Run a Node.js-based web server with a single command so that you can easily run the application from the application root directory without any special configuration.

Install Node.js from http://nodejs.org/^{Image Removed}.
We have created a Brunch-based application and checked it into Github. To check out the project from the Github repository, run:
git clone git@github.com:hortonworks/ambari
cd ambari-web
sudo npm install -g brunch
npm install

To run the web server (and to enable continuous build/deploy), run:
brunch watch --server
Or use the shorthand: brunch w --s

IDE: PhpStorm

All front-end developers should use PhpStorm by JetBrains.
JetBrains has granted all contributors of Apache Ambari a free license for PhpStorm and IntelliJ (if you also need to do Java development - IntelliJ is a superset of PhpStorm, but PhpStorm is lighter so it is recommended if you are not modifying Java code).

IDE Plugins

Go to Preferences->Plugins->Browse repositories and install “Node.js” and “Handlebars” plugins.
Coding Conventions

Before checking in any HTML/JavaScript/CSS files, they must be formatted with the IDE to maintain consistency.
Also, the IDE will give warnings in the editor about implicit globals, etc. Fix these warnings before checking in code.
IDE Code Formatting Customization

We will use all default settings for Code Style in the IDE, except for the following:
Go to Preferences
Code Style->General
Line separator (for new files): Unix
Make sure “Use tab character” is NOT checked
Tab size: 2
Indent: 2
Continuation indent: 2
Code Style->JavaScript:
Tabs and Indents
Make sure “use tab character” is NOT checked
Set Tab size, Indent, and Continuation indent to “2”.
Spaces->Other
Turn on “After name-value property separator ‘:’

In general, the following conventions should be followed for all JavaScript code: http://javascript.crockford.com/code.html^{Image Removed}

Exceptions to the rule from the above:
We use 2 spaces instead of 4.
Variable Declarations:
“It is preferred that each variable be given its own line and comment. They should be listed in alphabetical order.”
Comment only where it makes sense. No need to do alphabetical sorting.
“JavaScript does not have block scope, so defining variables in blocks can confuse programmers who are experienced with other C family languages. Define all variables at the top of the function.”
This does not need to be followed.

Unit Testing

Unit tests are written using Mocha.
Run unit tests via
brunch test

Coding Guidelines for Ambari Backend

Coding style

Following points are borrowed from hadoop wiki:
All public classes and methods should have informative Javadoc comments.
Do not use @author tags.
Code must be formatted according to Sun's conventions, with one exception:
Indent two spaces per level, not four.
Contributions must pass existing unit tests.
The code changes must be accompanied by unit tests. In cases where unit tests are not possible or don’t make sense an explanation should be provided on the jira.
New unit tests should be provided to demonstrate bugs and fixes. JUnit (junit4) is our test framework:
You must implement a class that uses @Test annotations for all test methods.
Define methods within your class whose names begin with test, and call JUnit's many assert methods to verify conditions. Please add meaningful messages to the assert statement to facilitate diagnostics.
By default, do not let tests write any temporary files to /tmp. Instead, the tests should write to the location specified by the test.build.data system property.
Logging levels should conform to Log4j levels
Use slf4j instead of commons logging as the logging facade.
Logger name should be the class name as far as possible.

Unit tests http://wiki.apache.org/hadoop/HowToDevelopUnitTests^{Image Removed}
The tests should be named *Test.java

Unit testing with databases
We should use JavaDB as the in-memory database for unit-test. The database layer/ORM should be configurable to use in memory database. Two things are important for the db in testing.
Ability to bootstrap the db with any initial data dynamically.
Ability to modify the db state out of band to simulate certain test cases.
One way to achieve the above could be to implement a database access layer only for testing purposes, but it might cause inconsistency in ORM objects, which needs to be figured out.

Stub Heartbeat handler
For testing purpose it may be a good idea to implement a stub heartbeat handler that only simulates interaction with the agents but doesn’t interact with any real agent:
It will expose an action queue similar to the real heartbeat handler, but will not send anything anywhere, will just periodically remove the action from the queue.
It will expose an interface to inject artificial responses for each of the actions, which can be used in tests to simulate agent responses.
It will also expose an interface to inject node state to simulate failure of nodes or lost heartbeats.
Guice framework can be used to inject stub heartbeat handler in testing.

EasyMock
EasyMock is our mocking framework of choice. It has been successfully used in hadoop.
A few important points:
An example of a scenario where Easymock is apt: Suppose we are testing deployment of a service but want to bypass a service dependency or want to inject an artificial component dependency, the dependency tracker object can be mocked to simulate the desired dependency scenario.
Ambari server is by and large a state driven system. EasyMock can be used to bypass the state changes and test components narrowly. However, it is probably better to rather use in-memory database to simulate state changes and use EasyMock only when certain behavior cannot be easily simulated. For example consider testing API implementation to get status of a transaction. It can be tested by mocking the action manager object, alternatively, it can be tested by setting the state in the in-memory database. In this case, the later is a more comprehensive test.
Avoid static methods and objects, Easymock cannot mock these. Use configuration or dependency injection to initialize static objects if they are likely to be mocked.
EasyMock cannot mock final classes, so those should be avoided for classes likely to be mocked.

Take a look at: http://www.easymock.org/EasyMock3_1_Documentation.html^{Image Removed} for docs.

Guice
Guice is a dependency injection framework and can be used to dynamically inject pluggable components.
Please refer
http://code.google.com/p/google-guice/wikJamiroquaii/Motivation^{Image Removed}
IMO we can use Guice in following scenarios:

Pluggable manifest generator: It may be desirable to have different implementation of manifest generator for non-puppet setup or for testing.
Injecting in-memory database (if possible) instead of a real persistent database for testing. It needs to be investigated how Guice fits with the ORM tools.
Injecting a stub implementation of heartbeat handler.
It may be a good idea to bind API implementations for management or monitoring via Guice. This will allow testing of APIs and the server independent of the implementation via mock implementations. For example, the management api implementation in coordinator can be mocked so that API definitions and URIs can be independently tested.
Injecting mock objects for dependency tracker, or stage planner for testing.