Page History

...

Marvin ↓ - our automation framework is a Python module that leverages the abilities of python and its multitude of libraries. Tests written using our framework use the unittest module under the hood. The unittest module is the Python version of the unit-testing framework originally developed by Kent Beck et al and will be familiar to the Java people in the form of JUnit. The following document will act as a tutorial introduction to those interested in testing CloudStack with python. This document does not cover the python language and we'll be pointing the reader instead to explore some tutorials that are more thorough on the topic. In the following we will be assuming basic python scripting knowledge from the reader. The reader is encouraged to walk through the steps after he/she has their environment setup and configured.

Environment

Developers

If you are a developer the cloudstack development environment is sufficient to get started

...

For Windows developers, it will be: easy_install tools/marvin/dist/Marvin-0.1.0.zip
This may fail while installing pycrypto. If it does, just install pycrypto binaries from here http://www.voidspace.org.uk/python/modules.shtml#pycrypto^{Image Removed} and re-run easy_install.

QA

If you are a QA engineer you won't need the entire codebase to build marvin.

Jenkins@builds.a.o holds artifacts of the marvin builds and you can download it.
The artifact (.tar.gz) is available after the build succeeds. Download it.
On the client machine where you will be writing/running tests from setup the following:
1. Install python 2.7 (http://www.python.org/download/releases/)
2. Install setuptools. Follow the instructions for your client machine (windows/linux/mac)
3. (for Windows only) Install pycrypto on Windows because windows does not bundle gcc to compile pycrypto.
The Marvin artifact you downloaded can now be installed. Any required python packages will be installed automatically
Code Block
easy_install tools/marvin/dist/Marvin-0.1.0.tar.gz

To test if the installation was successful get into a python shell

Code Block

root@cloud:~/cloudstack-oss/tools/marvin/dist# python
Python 2.7.1+ (r271:86832, Apr 11 2011, 18:05:24)
[GCC 4.5.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import marvin
>>> from marvin.cloudstackAPI import *

imports should happen without reporting errors.

First Steps

In our first steps we will build a simple API call and fire it against a CloudStack management server that is already deployed, configured and ready to accept API calls. You can pick any management server in your lab that has a few VMs running on it.Create a sample json config file telling us where your management server and database server are. Here's a sample:

...

The response is presented to us the way our UI receives it, as a JSON object. The object comprises of a list of configurations, each configuration showing the detailed dictionary (key, value) pairs of each config setting.

Putting it together

Listing stuff is all fine and dandy you might say - How do I launch VMs using python? And do I use the shell each time I have to do this? Well clearly not, we can have all the steps compressed into a python script. This example will show such a script which will:

...

Congratulations, your test has passed!

Advanced Example

We do not know for sure that the CentOS VM deployed earlier actually started up on the hypervisor host. The API tells us it did - so Cloudstack assumes the VM is up and running, but did the hypervisor successfully spin up the VM? In this example we will login to the CentOS VM that we deployed earlier using a simple ssh client that is exposed by the test framework. The example assumes that you have an Advanced Zone deployment of Cloudstack running. The test case is further simplified if you have a Basic Zone deployment. It is left as an exercise to the reader to refactor the following test to work for a basic zone.

...

Observe that unlike the previous test class - TestDeployVM - we do not have methods setUp and tearDown. Instead, we have the methods setUpClass and tearDownClass. We do not want the initialization (and cleanup) code in setup (and teardown) to run after every test in the suite which is what setUp and tearDown will do. Instead we will have the initialization code (creation of account etc) done once for the entire lifetime of the class. This is accomplished using the setUpClass and tearDownClass classmethods. Since the API client is only visible to instances of cloudstackTestCase we expose the API client at the class level using the getClsTestClient() method. So to get the API client we call the parent class (super(TestSshDeployVm, cls)) ie cloudstackTestCase and ask for a class level API client.

Test Pattern

An astute reader would by now have found that the following pattern has been used in the tutorial's test examples:

...

This pattern is useful to contain the entire test into one atomic piece. It helps prevent tests from becoming entangled in each other ie we have failures localized to one account and that should not affect the other tests. Advanced examples in our basic verification suite are written using this pattern. Test engineers are encouraged to follow the same unless there is good reason not to do so.

User Tests

The test framework by default runs all its tests under 'admin' mode which means you have admin access and visibility to resources in cloudstack. In order to run the tests as a regular user/domain-admin - you can apply the @UserName decorator which takes the arguments (account, domain, accounttype) at the head of your test class. The decorator will create the account and domain if they do not exist. Do NOT apply the decorator to a test method.

An example can be found at: cloudstack-oss/tools/testClient/testcase/test_userDecorator.py

Debugging & Logging

using the pydev plugin/ pdb and the testClient logs

...

While debugging with the PyDev plugin you can also place breakpoints in Eclipse for a more interactive debugging session.

Deployment Configuration

Marvin can be used to configure a deployed Cloudstack installation with Zones, Pods and Hosts automatically in to Advanced or Basic network types. This is done by describing the required deployment in a hierarchical json configuration file. But writing and maintaining such a configuration is cumbersome and error prone. Marvin's configGenerator is designed for this purpose. A simple hand written python description passed to the configGenerator will generate the compact json configuration of our deployment.

Examples of how to write the configuration for various zone models is within the configGenerator.py module in your marvin source directory. Look for methods describe_setup_in_advanced_mode()/ describe_setup_in_basic_mode().

What does it look like?

Below is such an example describing a simple one host deployment:

...

Notice that we didn't pass the -l option to deployAndRun. The reason being we don't want to just load the configuration but also deploy the configuration. This is the default behaviour of Marvin wherein the cloud configuration is deployed and the tests in the directory "tests/" are run against it.

How do I generate it?

The above one host configuration was described as follows:

...

This will deploy your cloud as given in the configuration file. Provided you have marvin installed on your machine from where you are running the above command and it can reach the required infrastructure

Sandbox Scripts

You don't always want to describe one hosts configurations in python files so we've included some common examples in the Marvin tarball under the sandbox directory. In the sandbox are configurations of a single host advanced and a single host basic zone that can be tailored to your environment using a simple properties file. The property file, setup.properties is contains editable name, value (name=value) pairs that you can change to the IPs, hostnames etc that you have in your environment. The properties file when passed to the python script will generate the JSON configuration for you.

...

Code Block

borderStyle	solid

root@cloud:~/incubator-cloudstack/tools/marvin/marvin/sandbox/advanced# python advanced_env.py -i setup.properties -o advanced.cfg
root@cloud:~/incubator-cloudstack/tools/marvin/marvin/sandbox/advanced# head -10 advanced.cfg
{
    "zones": [
        {
            "name": "Sandbox-XenServer",
            "guestcidraddress": "10.1.1.0/24",

... <snip/> ...

Marvin Nose Plugin

Nose extends unittest to make testing easier. Nose comes with plugins that help integrating your regular unittests into external build systems, coverage, profiling etc. Marvin comes with its own nose plugin for this so you can use nose to drive CloudStack tests. The plugin is installed on installing marvin. Running nosetests -p will show if the plugin registered successfully.

...

advanced - Typical Advanced Zone
basic - a basic zone without security groups
sg - a basic zone with security groups
eip - an elastic ip basic zone
advancedns - advanced zone with a netscaler device
devcloud - tests that will run only for the basic zone on a devcloud setup done using tools/devcloud/devcloud.cfg
speed = 0/1/2 (greater the value lesser the speed)
multihost/multipods/mulitcluster (test requires multiple set of hosts/pods/clusters)

Running Devcloud Tests

Some tests have been tagged to run only for devcloud environment. In order to run these tests you can use the following command after you've setup your management server and the host only devcloud is running with devcloud.cfg as its deployment configuration. This assumes you have the marvin-nose plugin installed on it as listed above.

No Format

~/workspace/cloudstack/incubator-cloudstack(branch:master*) » nosetests --with-marvin --marvin-config=tools/devcloud/devcloud.cfg --load -a tags='devcloud' test/integration/smoke

Test Deploy Virtual Machine ... ok
Test Stop Virtual Machine ... ok
Test Start Virtual Machine ... ok
Test Reboot Virtual Machine ... ok
Test destroy Virtual Machine ... ok
Test recover Virtual Machine ... ok
Test destroy(expunge) Virtual Machine ... ok

----------------------------------------------------------------------

Ran 7 tests in 0.001s

OK

Guidelines to choose scenarios for integration

There are a few do's and don'ts in choosing the automated scenario for an integration test. These are mostly for the system to blend well with the continuous test infrastructure and to keep environments pure and clean without affecting other tests.

Scenario

Every test should happen within a CloudStack test account. The order in which you choose the type of account to test within should be:

User > DomainAdmin > Admin

At the end of the test we delete this account so as to keep tests atomic and contained within a tenant's users space.

Checkin Tests

These are tests that can be run if you have a development environment of CloudStack. Light-Weight tests such as these can be run using the simulator as the hypervisor. The run time for tests is short and you should ensure that you have run them before making a commit that could potentially break basic functionality. Instructions for setting up and running these tests is explained here.

Check-In tests are the same as any other tests written using Marvin. The only additional step you need to do is ensure that your test is driven entirely by the API only. This makes it possible to run the test on a simulator. Once you have your test, you need to tag it t run on the simulator so the marvin test runner can pick it up during the checkin-test run.

For eg:

Code Block
@attr(tags =["simulator", "advanced", "smoke"]) def test_deploy_virtualmachine(self): """Tests deployment of VirtualMachine """

More Examples

Examples of tests with more backend verification and complete integration of suites for network, snapshots, templates etc can be found in the test/integration/smoke and test/integration/component. Almost all of these test suites use common library wrappers written around the test framework to simplify writing tests. These libraries are part of marvin.integration. Ensure that you've gone through the existing tests related to your feature before writing your own.

The integration library takes advantage of the fact that every resource - VirtualMachine, ISO, Template, PublicIp etc follows the pattern of

create - where we cause creation of the resource eg: deployVirtualMachine
delete - where we delete our resource eg: deleteVolume
list - where we look for some state of the resource eg: listPods

Marvin can auto-generate these resource classes using API discovery. The auto-generation ability is being added as part of this refactor

Guidelines to choose scenarios for integration

There are a few do's and don'ts in choosing the automated scenario for an integration test. These are mostly for the system to blend well with the continuous test infrastructure and to keep environments pure and clean without affecting other tests.

Scenario

Every test should happen within a CloudStack test account. The order in which you choose the type of account to test within should be:

User > DomainAdmin > Admin

At the end of the test we delete this account so as to keep tests atomic and contained within a tenant's users space.

All tests must be written with the perspective of the API. UI directions are often confusing and using the rich API often reveals further test scenarios. You can capture the API arguments using cloudmonkey/firebug.
Tests should be generic enough to run in any environment/lab - under any hypervisor.** If this is not possible then it is appropriate to mark the test with an @attr attribute to signify any specifics. eg: @attr(hypervisor='vmware') for runs only on vmware
Every resource should be creatable in the test from scratch
All tests must be written with the perspective of the API. UI directions are often confusing and using the rich API often reveals further test scenarios. You can capture the API arguments using cloudmonkey/firebug.
Tests should be generic enough to run in any environment/lab - under any hypervisor.** If this is not possible then it is appropriate to mark the test with an @attr attribute to signify any specifics. eg: @attr(hypervisor='vmware') for runs only on vmware
Every resource should be creatable in the test from scratch.** referring to an Ubuntu template is probably not a good idea. Your test must show how to fetch this template or give a static location from where the test can fetch it.
Do not change global settings configurations in between a test. Make two separate tests for this. All tests run against one given deployment and altering the settings midway is not effective

Backend Verification with paramiko/and other means

Verifying status of resources within hypervisors is fine. Most hypervisors provide standard SSH server access
Your tests should include the complete command and its expected output. ** e.g. iptables -L INPUT -# to list the INPUT chain of iptables
If you execute multiple commands then all of them should be chained together on one line** e.g: service iptables stop; service iptables start #to stop and start iptables
Your script must execute over ssh because this is how Marvin will execute it** ssh <target-backend-machine> "<your script>" #should return the output of your script** move the credential specific information into the deployment config file and/or use a standard credential
Most external devices like F5/ NetScaler have ssh open to execute commands. But you must include the command and its expected output in the test as not everyone is aware of the device's CLI
If you are using a UI like vCenter to verify something most likely you cannot automate what you see there because there are no ASLv2 licensed libraries for vmware/esx as of today.

Python Resources

The single largest python resource is the python website itself - http://www.python.org
Mark Pilgrim's - "Dive Into Python" - is another great resource. The book is available for free online - http://www.diveintopython.net. Chapter 1- 6 cover a good portion of language basics and Chapter 13 & 14 are essential for anyone doing test script development
To read more about the assert methods the language reference is the ideal place - http://docs.python.org/library/unittest.html.

More Examples

Examples of tests with more backend verification and complete integration of suites for network, snapshots, templates etc can be found in the test/integration/smoke directory. Almost all of these test suites use common library wrappers written around the test framework to simplify writing tests. These libraries are part of marvin.integration. You may start using these libraries at your convenience but there's no better way than to write the complete API call yourself to understand its behaviour.

The libraries take advantage of the fact that every resource - VirtualMachine, ISO, Template, PublicIp etc follows the pattern of

...

. You can also use Zed Shaw's book, Learn Python the Hard Way, if programming is something completely new to you
To read more about the assert methods the language reference is the ideal place - http://docs.python.org/library/unittest.html.

...

Acknowledgements

The original author of the testing framework - Edison Su
Maintenance and bug fixes - Prasanna Santhanam
Documentation - Prasanna and Edison

...

Space shortcuts

Child pages

Versions Compared

Old Version 42

New Version 43

Key

Environment

Developers

QA

First Steps

Putting it together

Advanced Example

Test Pattern

User Tests

Debugging & Logging

Deployment Configuration

What does it look like?

How do I generate it?

Sandbox Scripts

Marvin Nose Plugin

Running Devcloud Tests

Guidelines to choose scenarios for integration

Scenario

Checkin Tests

More Examples

Guidelines to choose scenarios for integration

Scenario

Backend Verification with paramiko/and other means

Python Resources

More Examples

Acknowledgements