Page History

...

DevCloud (XenServer - basic)

The XCP-based DevCloud is is capable of deploying a basic zone CloudStack environment. We've patched this appliance with a clone of the incubator-CloudStack ASF repo and our test utilities (marvin, cloudmonkey) and created a 'testcloud'   appliance. The following workflow explains the way tests are run within the appliance:

Workflow

The testcloud  appliance appliance contains an init script at /etc/init.d/DevCloud that will start up a testrunner process when the appliance boots. Following this the testrunner will perform the necessary setup to prepare a basic zone CloudStack management server.

1. The test runner fetches the latest HEAD (currently set to master) from the git repository and builds CloudStack.  In In other words - mvn install
2. This is followed by an installation of marvin so we have the latest API's available via the test framework
3. Further on the client inside a jetty:run will start up a management server process and have the API server listening for requests
4. We then configure this management server with the basic zone DevCloud configuration with one XCP host and local storage. i.e., the config at tools/DevCloud/DevCloud.cfg
5. Once we have the basic zone running we will do a basic health check that verifies the following
   1. All system VM's have come up and are successfully 'Running'
   2. All the featured templates (built-in) are in Ready (downloaded) state
6. If the health check succeeds the environment is ready to run tests
7. Wiki Markup[nose|https://nose.readthedocs.org/en/latest/]'s testrunner with the marvin plugin will collect all the tests marked with the '_ attribute '_ of '_DevCloud'  \[@attr(tags='DevCloud')\] , _ put them in a suite and run them
   1. The nose xUnit plugin gives us the XML output for consumption into jenkins
8. Management Server log, logs from the test appliance's init script and the test results are posted to the machine hosting the 'testcloud' appliance and held as artifacts on the jenkins job for troubleshooting and held as artifacts on the jenkins job for troubleshooting
9. A jenkins slave agent within the host will send through the results and logs to the jenkins master at jenkins.c.o

...

The host machine is a Ubuntu server with VirtualBox 4.2 installed with guest additions. VirtualBox does performs DHCP for the 'testcloud' workers that it spins up. The testcloud image itself is cloned into multiple test workers (currently 5). A simple scheduler will pick up an idle worker VM on trigger from jenkins master that is polling for commits on the git:repo. Once picked up each worker will perform all the tests and post the results back to the gateway/DHCP which in our case is the host machine running VirtualBox. A jenkins slave runs in headless mode on the host machine and posts results back to jenkins master.

After the testworker has done its job and posted its results back to jenkins we wait for a timeout (1800s) to release it back to the pool of workers. To clean up the image of any remnants from the previous test run the worker is restored to a base snapshot keeping the environment clean for the subsequent run.

Code
When we change mvn instructions for building CloudStack and/or the tests that need to run - the test runner testrunner process will fetch the latest version of itself from a github repo running DevCloud-ci. This is useful so useful so we don't change the image of the testcloud appliance when our build process changes.

...

The jenkins job is a vanilla job  enabled enabled with JUnit reports. Email E-mail will be enabled to CloudStack-commits@i.a.o

...

Following additional packages are installed:  VirtualBox VirtualBox 4.2.4,  VirtualBox VirtualBox 4.2.4 - Extension Pack,  VirtualBox VirtualBox Guest additions

1. create a jenkins user  on on this machine that will run a jenkins-slave service which will interact with jenkins master to trigger the test runs and send test reports.  youyou'll need access to add nodes and create jobs on the jenkins CI. once the node is setup and the slave is connected you are ready to schedule jobs.

2. modify the default network in VirtualBox to perform DHCP as follows and enable the DHCP server

...

and change the password in the /etc/init.d/DevCloud init script script

28       python $SCRIPTDIR/setUpTestWorker.py --host 192.168.56.1 --user jenkins --pass yourpassword --out /var/lib/jenkins/reports

and then reinitialize the system-V service

...

$ vboxmanage controlvm testcloud poweroff

$ vboxmanage export testcloud --output testcloudworker.ova

6. import the appliance in to virtual box again but creating as many workers as you need. 

$for i in {1..5}; do vboxmanage import testcloudworker.ova --vsys 0 --eula accept --vsys 0 --vmname testworker$i; done

7.  create create a base snapshot that will be restored to after the testworker VM has finished running a test. This is to restore the appliance to its clean state for the next test run

...

jenkins@apache-81-1:~$ vboxmanage list vms
"testworker1" {e6b5924b-efac-40cf-9621-3bb0b4683ab9}
"testworker2" {070da3d1-4e09-4171-9ec0-17c10d8ca425}
"testworker3" {dff3a63c-32ac-4631-aae1-66837a049fb0}
"testworker4" {931baa82-1b37-460e-a500-7764a3ee38dd}
"testworker5" {0bdd381d-bfc0-4b8e-8bbc-908fbe7b4710}

TODO's and FIXME's

1. Speed up the mvn clean install step with a local nexus proxy/maven cache. Currently this eats up around 5m

...

3. Draft a proposal for moving this to builds.a.o

DevCloud (KVM - advanced)

WIP: We intend to do the same for a KVM based DevCloud environment to so as to test any functionality that is essential for advanced zone CS CloudStack deployments.