Page History

Why Testing of a Release Artifact Matters

For an apache project, a VOTE on a release candidate is a very important process.  By voting (particularly for PPMC PMC members and committers), you are saying to the world that "yes, I have download, verified and tested using the project's procedure for testing".  Your +1, 0 or -1 vote is an indication of the success of the steps listed.  The more time that each individual spends reviewing the artifacts, the higher confidence we can have in both the release itself and our ability to pass an IPMC vote later on.  Remember, we, as a podling, are going to be held to a higher standard for our releases!  The IPMC (and our mentors) are looking for us to show that we understand what it means to have a high quality release.

Test with source code release

To setup an environment for CloudStack, usually it will take a while, e.g install a hypervisor host, install Mysql server etc. Here we'll use DevCloud2 to simplify the test procedure.  

Setup environment

Install the devcloud VM image.

• Install VirtualBox(https://www.virtualbox.org/wiki/Downloads) on your machine.
  • Note: If you are on Mac OS X(10.6 or below), must enable 64bit(http://support.apple.com/kb/HT3773)
• Download DevCloud image from http://download.cloud.com/templates/devcloud/DevCloud.ova into your machine.
• Import DevCloud.ova into VirtualBox:
  • Open VirtualBox, File->Import Appliance->choose DevCloud.ova
  • NOTE: Change the memory to 2046
  • NOTE: please do NOT select "re-initialize the MAC address of all network cards"
• Start devcloud VM
• For more information about how to install devcloud, please refer to http://wiki.cloudstack.org/display/COMM/DevCloud

Install your required development tools (if not already available)

Setup Pre-requisites on an Ubuntu 12.04 or Mint 13 machine

sudo aptitude install openjdk-6-jdk ant
sudo apt-get update
sudo apt-get install python-software-properties
sudo add-apt-repository ppa:natecarlson/maven3
sudo aptitude update
sudo aptitude install maven3

Install JSch:

release.

Test with source code release

To setup an environment for CloudStack, usually it will take a while, e.g install a hypervisor host, install Mysql server etc. Here we'll use DevCloud2 to simplify the test procedure.  

Setup environment

Install the DevCloud VM image via the instructions on the DevCloud wiki page.
Install your required development tools (if not already available)

Setup Pre-requisites on an Ubuntu 12.04 or Mint 13 machine

sudo aptitude install openjdk-6-jdk ant
sudo aptitude update
sudo aptitude install python-software-properties
sudo add-apt-repository ppa:natecarlson/maven3
sudo aptitude update
sudo aptitude install maven3

Install JSch:

wget http://downloads.sourceforge.net/project/jsch/jsch.jar/0.1.48/jsch-0.1.48.jar
sudo cp jsch-0.1.48.jar /usr/share/ant/lib/

Install Boto (assumes you have Python and Setup Tools installed):

sudo easy_install boto

Setup Pre-requisites on OSX

OSX users should be able to use their default JVM for this testing.

OSX users need to have Apache Ant and Maven 3 installed.

Ant may already be on your system. Run which ant to see. If not, Ant can be installed via MacPorts or from source.

Maven 3 can be installed with Homebrew, or can be downloaded and installed from the Maven homepage.

Next, you will need to install OS X tools:

• Install XCode from the AppStore (Xcode is around 1.53 GB, so it will take a while before this completes. Look at the "Purchases" window of the App Store to view the download progress)
  • On Mac OS X, installation of Xcode via App Store doesn't actually install Xcode - you will need to go to Finder -> Applications -> Xcode and then you will be prompted to install Xcode.
  • After the Xcode installation goes through, you will need to install the Command Line Tools for Xcode. To do this, open Xcode, and then, on the top left corner of the screen, click on Xcode -> Preferences. In the dialog box that opens, click on Downloads. There, click on Install for Command Line Tools.

Then you will need cdrtools, which is available as package in both Homebrew and MacPorts.

Next, you will need to install JSch.

First download the binary distribution of Ant.

Unpack the archive, and run:

sudo cp lib/ant-jsch.

wget http://downloads.sourceforge.net/project/jsch/jsch.jar/0.1.48/jsch-0.1.48.jar
sudo cp jsch-0.1.48.jar /usr/share/ant/lib/

Install Boto (assumes you have Python and Python Setup Tools already installed):

sudo easy_install boto

Setup Pre-requisites on OSX

OSX users should be able to use their default JVM for this testing.

OSX users need to have Apache Ant and Maven 3 installed.

Ant may already be on your system. Run which ant to see. If not, Ant can be installed via MacPorts or from source.

Maven 3 can be installed with Homebrew, or can be downloaded and installed from the Maven homepage.

Next, you will need to install OS X tools:

• Install XCode from the AppStore (Xcode is around 1.53 GB, so it will take a while before this completes. Look at the "Purchases" window of the App Store to view the download progress)
  • On Mac OS X, installation of Xcode via App Store doesn't actually install Xcode - you will need to go to Finder -> Applications -> Xcode and then you will be prompted to install Xcode.
  • After the Xcode installation goes through, you will need to install the Command Line Tools for Xcode. To do this, open Xcode, and then, on the top left corner of the screen, click on Xcode -> Preferences. In the dialog box that opens, click on Downloads. There, click on Install for Command Line Tools.

Then you will need cdrtools, which is available as package in both Homebrew and MacPorts.

Next, you will need to install JSch.

First download the binary distribution of Ant.

Unpack the archive, and run:

sudo cp lib/ant-jsch.jar /usr/share/ant/lib

Install Boto (assumes you have Python and Python Setup Tools already installed):

sudo easy_install boto

Test the Build

Clean up your environment

rm -Rf /tmp/cloudstack
rm -Rf ~/.m2

Download the source build

Download the artifacts:

Test the Build

Clean up your environment

rm -Rf /tmp/cloudstack
rm -Rf ~/.m2

Download the source build

Download the artifacts:

mkdir /tmp/cloudstack; cd /tmp/cloudstack
wget --no-check-certificate https://dist.apache.org/repos/dist/release/cloudstack/KEYS
wget --no-check-certificate https://dist.apache.org/repos/dist/dev/cloudstack/4.3.0/apache-cloudstack-4.3.0-src.tar.bz2
wget --no-check-certificate https://dist.apache.org/repos/dist/dev/cloudstack/4.3.0/apache-cloudstack-4.3.0-src.tar.bz2.asc
wget --no-check-certificate https://dist.apache.org/repos/dist/dev/cloudstack/4.3.0/apache-cloudstack-4.3.0-src.tar.bz2.md5

mkdir /tmp/cloudstack; cd /tmp/cloudstack
wget --no-check-certificate https://dist.apache.org/repos/dist/release/cloudstack/KEYS
wget --no-check-certificate https://dist.apache.org/repos/dist/devdev/cloudstack/4.13.0/apache-cloudstack-4.13.0-src.tar.bz2
wget --no-check-certificate https://dist.apache.org/repos/dist/dev/cloudstack/4.1.0/apache-cloudstack-4.1.0-src.tar.bz2.asc
wget --no-check-certificate https://dist.apache.org/repos/dist/dev/cloudstack/4.1.0/.sha

Install gpg (if needed): sudo apt-get install gpg

Import keys:  

This can be done by importing the keys stored in the source distribution's KEYS file:  gpg --import KEYS

Alternatively, you could download the signing keys (ID's found in the KEYS file) individually via a keyserver.  Ex:  gpg --recv-keys CC56CEA8

Verify signatures and hash files:

gpg --verify apache-cloudstack-4.13.0-src.tar.bz2.md5
wgetasc

This command should return "Good Signature".

gpg --no-check-certificate https://dist.apache.org/repos/dist/dev/cloudstack/4.1.0/print-md MD5 apache-cloudstack-4.3.0-src.tar.bz2 | diff - apache-cloudstack-4.13.0-src.tar.bz2.shamd5

Install gpg (if needed): sudo apt-get install gpg

Import keys:  

This can be done by importing the keys stored in the source distribution's KEYS file:  gpg --import KEYS

Alternatively, you could download the signing keys (ID's found in the KEYS file) individually via a keyserver.  Ex:  gpg --recv-keys CC56CEA8

Verify signatures and hash files:

gpg --verifygpg --print-md SHA512 apache-cloudstack-4.3.0-src.tar.bz2 | diff - apache-cloudstack-4.13.0-src.tar.bz2.ascsha

This command Each of these commands should return "Good Signature".

gpg --print-md MD5 apache-cloudstack-4.1.0-src.tar.bz2 | diff - apache-cloudstack-4.1.0-src.tar.bz2.md5
gpg --print-md SHA512 apache-cloudstack-4.1.0-src.tar.bz2 | diff - apache-cloudstack-4.1.0-src.tar.bz2.sha

Each of these commands should return no output. If there is any output from them, then there is a difference between the hash you generated locally and the hash that has been pulled from the server.

Extract the source code and verify the contents:

Get the commit hash from the VOTE email (ex: 4cd60f3d1683a3445c3248f48ae064fb573db2a1). The value will change between releases.

Create two new temporary directories:

mkdir /tmp/cloudstack/git
mkdir /tmp/cloudstack/tree

Pull down the git repo:

git clone https://git-wip-us.apache.org/repos/asf/cloudstack.git /tmp/cloudstack/git
cd /tmp/cloudstack/git
git archive --prefix=/tmp/cloudstack/tree/ <commit-hash> | tar Pxf -

Unpack the release artifact:

cd /tmp/cloudstack
tar xvfj apache-cloudstack-4.1.0-src.tar.bz2

Compare the contents of the release artifact with the contents pulled from the repo:

diff -r /tmp/cloudstack/apache-cloudstack-4.1.0-src /tmp/cloudstack/tree

Nothing should be different.

Verify the Code's License Headers

cd /tmp/cloudstack/apache-cloudstack-4.1.0-src
mvn --projects='org.apache.cloudstack:cloudstack' org.apache.rat:apache-rat-plugin:0.8:check

The build should FAIL if there are any non-compliant files that are not specifically excluded from the ASF license header requirement. You can optionally review the target/rat.txt file after the run completes. If the build passes, RAT is saying that we are compliant and this test passes.

Compile

Note: If you're on Ubuntu and using the PPA:natecarlson/maven3 (viz. Installing tools above), you've to use mvn3 instead of mvn, so mvn3 -P deps

mvn clean install

Start CloudStack management server

ant rdebug

Configure CloudStack management server

• Access web UI: http://localhost:8080/client
• Login with admin/password
• Choose "I have used cloudstack before, skip this guide"
• Click "Infrastructure", click the "View All" button in the "Zones" display box, click "add zone"
• Select "basic zone" during the add zone wizard.
• Type name "test cloud", dns1 "8.8.8.8", internal dns1 "10.0.2.3", hypervisor "Xenserver", select "local storage enabled" (Say Yes if prompted to agree), then click next.
• Skip "physical network setup" page
• In "pod" page, type: pod name: 'test pod', reserved system gateway: 10.0.2.2, reserved system mask: 255.255.255.0, start reserved system ip: 10.0.2.200, end reserved system ip: 10.0.2.220
• In "guest traffic" page, guest gateway: 10.0.2.2, netmask: 255.255.255.0, guest start ip: 10.0.2.100, guest end ip: 10.0.2.199
• In "cluster" page, type cluster name: "test cluster"
• In "host " page, host name: 10.0.2.15, user name: root, password: password
• In "secondary storage" page, server: 10.0.2.15, path: /opt/storage/secondary
• In "zone creation comple", select "yes" to enable zone.

Wait for secondary storage VM coming up.

Depending on your local system configuration, and available RAM / Disk IO / Net IO, this step will take varied amounts of time.  It's reasonable to expect 30 minutes (or more), but it could be less.

You can tell if the SSVM has been started by navigating to the Infrastructure page, and clicking on System VMs.  There should be 2 listed, and their status needs to be displayed as "Started".

Add a new instance

• In the first step of "Add instance " wizard ,  select template,
• In step 2, select "tiny Linux"
• In step 3, select "tinyOffering"
• In step 4, select "No thanks"
• Skip step 5 and 6.
• Launch VM

Set some Global Settings

In the UI, set the following global settings:

• expunge.interval = 120
• expunge.delay = 120
• enable.ec2.api = true

Now CTL-C your rdebug, and then "ant rdebug" again (to restart the management server).  This step is critical!

Additional Testing

Go to the instances list, click on the name of your new instance, and then click on the "show console" button.  The console of the VM should appear in a new window.

Stop the instance

Destroy the instance.

Wait for at least 2 minutes for the instance to be expunged (you can confirm that it has, by ssh'ing into devcloud and running xe vm-list to see if it's no longer listed).

Destroying the instance is important, and confirming that it was expunged prior to trying to test the EC2 API. This is due to the resource constraints on the DevCloud image.

EC2 testing

• Under Accounts, generate keys for the admin user in the admin account (save these for future use)
• Under service offerings, rename tinyOffering as m1.small
• In your Virtual Box settings, forward port 7080 to local port 7080
• In your local machine, cd into the awsapi-setup/setup folder of the source
• Use the python script cloudstack-aws-api-register to register the user. Like so:

python cloudstack-aws-api-register --apikey=JdTxDI2rOZ0RDZdgF1O_ZxfISxNuema9gQ3NyE4z5psX4sgSxE4ueEBohG3bfqWHr7BYP6_sAghXTIvm3noqkw --secretkey=fVT0r5lETIJThRGyec-Eg4WFm9_IN5tNIut7Ac5BJW0VExFGdRZEqA_JFvxdTFDH2ZeRcbO25xPQcyxt0VRWKg --cert=/tmp/cert.pem --url=http://localhost:7080/awsapi

In case you don't have a X509 ssl certificate, this is how you can generate one:

openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /path/to/private_key.pem -out /path/to/cert.pem

• Use a python script similar as below (change the keys) to launch an instance:

#!/usr/bin/env python

import sys
import os
import boto
import boto.ec2

region = boto.ec2.regioninfo.RegionInfo(name="ROOT",endpoint="localhost")
apikey='JiKIBbp6GRe3-7Ma-KF_pJl69BAsv2smJEA3So4DLfG0JCa9u5VnGeS5qsSo6cHiArzu8pRwEr4DkGy6M5inTw'
secretkey='x9iMD9XY0xsnLy_1EZQi6lOXAY5hH-O6S2z_VFVTWMO5_GAIdeSm93tNk09rb56cB1bAQKZ0vSOztBrusZRu6g'

def main():
        '''Establish connection to EC2 cloud'''
        conn =boto.connect_ec2(aws_access_key_id=apikey,
                       aws_secret_access_key=secretkey,
                       is_secure=False,
                       region=region,
                       port=7080,
                       path="/awsapi",
                       api_version="2010-11-15")

        '''Get list of images that I own'''
        images = conn.get_all_images()
        print images
        myimage = images[0]
        '''Pick an instance type'''
        vm_type='m1.small'
        reservation = myimage.run(instance_type=vm_type,security_groups=['default'])

if __name__ == '__main__':
        main()

...

no output. If there is any output from them, then there is a difference between the hash you generated locally and the hash that has been pulled from the server.

Extract the source code and verify the contents:

Get the commit hash from the VOTE email (ex: 4cd60f3d1683a3445c3248f48ae064fb573db2a1). The value will change between releases.

Create two new temporary directories:

mkdir /tmp/cloudstack/git
mkdir /tmp/cloudstack/tree

Pull down the git repo:

git clone https://git-wip-us.apache.org/repos/asf/cloudstack.git /tmp/cloudstack/git
cd /tmp/cloudstack/git
git archive --format=tar --prefix=/tmp/cloudstack/tree/ <commit-hash> | tar Pxf -

Unpack the release artifact:

cd /tmp/cloudstack
tar xvfj apache-cloudstack-4.3.0-src.tar.bz2

Compare the contents of the release artifact with the contents pulled from the repo:

diff -r /tmp/cloudstack/apache-cloudstack-4.3.0-src /tmp/cloudstack/tree

Nothing should be different.

Verify the Code's License Headers

cd /tmp/cloudstack/apache-cloudstack-4.3.0-src
mvn --projects='org.apache.cloudstack:cloudstack' org.apache.rat:apache-rat-plugin:0.8:check

The build should FAIL if there are any non-compliant files that are not specifically excluded from the ASF license header requirement. You can optionally review the target/rat.txt file after the run completes. If the build passes, RAT is saying that we are compliant and this test passes.

Compile and Start Management Server

Note: If you're on Ubuntu and using the PPA:natecarlson/maven3 (viz. Installing tools above), you've to use mvn3 instead of mvn, so mvn3 -P deps

mvn -P developer,systemvm clean install
mvn -P developer -pl developer,tools/devcloud -Ddeploydb
mvn -pl :cloud-client-ui jetty:run

Configure CloudStack management server

Once the management server starts on your local machine, execute the following commands to bring up a basic zone using the devcloud2 VM:

Deploy DevCloud (make sure mysql-connector-python is installed and that the management server is running)

$ mvn -P developer -pl tools/devcloud -Ddeploysvr

Or, if the above does not work, maybe you're running mvn in debug mode using some MAVEN_OPTS, try marvin:

$ cd tools/devcloud; python ../marvin/marvin/deployDataCenter.py -i devcloud.cfg

The above will deploy a zone with settings defined in tools/devcloud/devcloud.cfg which sets some global settings and will take some time. After this, you should restart management server and destroy any system vms which may have started for the global settings to take effect.

Validate the UI

Access web UI: http://localhost:8080/client, and login with admin/password. You can tell if the SSVM has been started by navigating to the Infrastructure page, and clicking on System VMs.  There should be 2 listed, and their status needs to be displayed as "Started".

Add a new instance:

• In the first step of "Add instance " wizard, select template,
• In step 2, select "tiny Linux"
• In step 3, select "tinyOffering"
• In step 4, select "No thanks"
• Skip step 5 and 6.
• Launch VM

Test other functionality, to suite your tastes (and perhaps using non-DevCloud infrastructure.

Validate AWSAPI interface

In a separate terminal run:

mvn -Pawsapi -pl :cloud-awsapi jetty:run

Via the UI:
1. Destroy any running instances
2. Rename the tinyoffering compute offering 'm1.small'

Then using the access key and secret key of your admin user run the following python script:

#!/usr/bin/env python

import boto
import boto.ec2

accesskey="2IUSA5xylbsPSnBQFoWXKg3RvjHgsufcKhC1SeiCbeEc0obKwUlwJamB_gFmMJkFHYHTIafpUx0pHcfLvt-dzw"
secretkey="oxV5Dhhk5ufNowey7OVHgWxCBVS4deTl9qL0EqMthfPBuy3ScHPo2fifDxw1aXeL5cyH10hnLOKjyKphcXGeDA"
 
region = boto.ec2.regioninfo.RegionInfo(name="ROOT", endpoint="localhost")
conn = boto.connect_ec2(aws_access_key_id=accesskey, aws_secret_access_key=secretkey, is_secure=False, region=region, port=7080, path="/awsapi", api_version="2012-08-15")

images=conn.get_all_images()
print images

res = images[0].run(instance_type='m1.small',security_groups=['default'])

If the keys have changed, using the above script verbatim won't work. Make sure to update the keys.

End!

If we made it this far, the release has at least been reasonably "smoke tested".