Page History

...

Marvin requires Python 2.6 for installation but the tests written using marvin utilize Python 2.7 to the fullest. You should follow the test environment setup instructions here before proceeding further.

Dependencies

• Marvin dependencies are installed automatically by the installer

• python-paramiko - remote ssh access, ssh-agent

• mysql-connector-python - db connectivity

• nose - test runner, plugins

• marvin-nose - marvin plugin for nose

• should_dsl - should assertions dsl-style

Windows specifics

• For Windows (non-cygwin) development environment, following additional steps are needed to be able to install Marvin:

• The mysql-connector-python for Windows can be found here

• pycrypto binaries are required since Windows does not have gcc/glibc. Download here

Compiling and Packaging

The developer profile compiles and packages Marvin. It does NOT install marvin your default PYTHONPATH.

...

features auto-completion for the test that follows. We will explain how to
run the tests in eclipse later.

• create a testcase class

• setUp a user account - name: user, passwd: password

• deploy a VM into that user account using the default small service offering and CentOS template

• verify that the VM we deployed reached the 'Running' state

• tearDown the user account - basically delete it to cleanup acquired resources

Here is our test_deploy_vm.py module:

python
#All tests inherit from cloudstackTestCase
from marvin.cloudstackTestCase import cloudstackTestCase

#Import Integration Libraries

#base - contains all resources as entities and defines create, delete, list operations on them
from marvin.integration.lib.base import Account, VirtualMachine, ServiceOffering

#utils - utility classes for common cleanup, external library wrappers etc
from marvin.integration.lib.utils import cleanup_resources

#common - commonly used methods for all tests are listed here
from marvin.integration.lib.common import get_zone, get_domain, get_template

class TestData(object):
    """Test data object that is required to create resources
    """
    def __init__(self):
        self.testdata = {
            #data to create an account
            "account": {
                "email": "test@test.com",
                "firstname": "Test",
                "lastname": "User",
                "username": "test",
                "password": "password",
            },
            #data reqd for virtual machine creation
            "virtual_machine" : {
                "name" : "testvm",
                "displayname" : "Test VM",
            },
            #small service offering
            "service_offering": {
                "small": {
                    "name": "Small Instance",
                    "displaytext": "Small Instance",
                    "cpunumber": 1,
                    "cpuspeed": 100,
                    "memory": 256,
                },
            },
            "ostype": 'CentOS 5.3 (64-bit)',
        }


class TestDeployVM(cloudstackTestCase):
    """Test deploy a VM into a user account
    """

    def setUp(self):
        self.testdata = TestData().testdata
        self.apiclient = self.testClient.getApiClient()

        # Get Zone, Domain and Default Built-in template
        self.domain = get_domain(self.apiclient, self.testdata)
        self.zone = get_zone(self.apiclient, self.testdata)
        self.testdata["mode"] = self.zone.networktype
        self.template = get_template(self.apiclient, self.zone.id, self.testdata["ostype"])

        #create a user account
        self.account = Account.create(
            self.apiclient,
            self.testdata["account"],
            domainid=self.domain.id
        )
        #create a service offering
        self.service_offering = ServiceOffering.create(
            self.apiclient,
            self.testdata["service_offering"]["small"]
        )
        #build cleanup list
        self.cleanup = [
            self.service_offering,
            self.account
        ]

    def test_deploy_vm(self):
        """Test Deploy Virtual Machine

        # Validate the following:
        # 1. Virtual Machine is accessible via SSH
        # 2. listVirtualMachines returns accurate information
        """
        self.virtual_machine = VirtualMachine.create(
            self.apiclient,
            self.testdata["virtual_machine"],
            accountid=self.account.name,
            zoneid=self.zone.id,
            domainid=self.account.domainid,
            serviceofferingid=self.service_offering.id,
            templateid=self.template.id
        )

        list_vms = VirtualMachine.list(self.apiclient, id=self.virtual_machine.id)

        self.debug(
            "Verify listVirtualMachines response for virtual machine: %s"\
            % self.virtual_machine.id
        )

        self.assertEqual(
            isinstance(list_vms, list),
            True,
            "List VM response was not a valid list"
        )
        self.assertNotEqual(
            len(list_vms),
            0,
            "List VM response was empty"
        )

        vm = list_vms[0]
        self.assertEqual(
            vm.id,
            self.virtual_machine.id,
            "Virtual Machine ids do not match"
        )
        self.assertEqual(
            vm.name,
            self.virtual_machine.name,
            "Virtual Machine names do not match"
        )
        self.assertEqual(
            vm.state,
            "Running",
            msg="VM is not in Running state"
        )

    def tearDown(self):
        try:
            cleanup_resources(self.apiclient, self.cleanup)
        except Exception as e:
            self.debug("Warning! Exception in tearDown: %s" % e)

Parts of the test

imports

• cloudstackTestCase - All the cloudstack marvin tests inherit from this class. The class provides the tests with apiclient and dbclients using which calls can be made to the API server. dbclient is useful for verifying database values using SQL

• integration.lib.base - the base module contains all the resources one can manipulate in cloudstack. eg: VirtualMachine, Account, Zone, VPC, etc. Each resource defines a set of operations. For eg: VirtualMachine.deploy, VirtualMachine.destroy, VirtualMachine.list, etc. Each operation makes an API call: For eg: VirtualMachine.recover -> recoverVirtualMachineCmd returning a recoverVirtualMachineResponse. When dealing with the tests one only has to identify the set of resources one will be using and all the operations will autocomplete if you are using an IDE like PyDev/PyCharm

• integration.lib.utils - utility classes for performing verification/actions outside the API. Eg. ssh utilities, random_string generator, cleanup etc.

• integration.lib.common - simple methods that are commonly required for most of the tests, eg: a template -get_template, a zone-get_zone, etc

TestData

The test data class carries information in a dictionary object. (key, value) pairs in this class are needed to be externally supplied to satisfy an API call. For eg: In order to create a VM one needs to give a displayname and the vm name. These are externally supplied data. It is not mandatory to use the testdata class to supply to your test. In all cases you can simply send the right arguments to the Resource.operation( method of your resource without using testdata dictionaries. The advantage of testdata is keeping all data to be configurable in a single place.

Although test data is at the top of our test class, it is only identified as and when we start writing our test. In our case we have identified that we need an account (firstname,lastname etc), a virtual_machine (with name and displayname) and a service_offering (with cpu: 128 and some memory) as test data.

TestDeployVM

• TestDeployVM is our test class that holds the suite of tests we want to perform. All test classes start with Capital caseing and the Test prefix. Ideally only one test class is contained in every module

• The comment in triple quotes is used to describe the scenario being tested.

• setUp() - the setup method is run before every test method in the class and is used to initialize any common data, clients, resources required in our tests. In our case we have initialized our testclients - apiclient and dbclient identified the zone, domain and template we will need for the VM and created the user account into which the VM shall be deployed into.

...

• self.cleanup

...

• =

...

• [

...

• ]

...

• .

...

• The

...

• cleanup

...

• list

...

• contains

...

• the

...

• list

...

• of

...

• objects

...

• which

...

• should

...

• be

...

• destroyed

...

• at

...

• the

...

• end

...

• of

...

• our

...

• test

...

• run

...

• in

...

• the

...

• tearDown

...

• method

• tearDown() - the teardown method simply calls the cleanup (delete) associated with every resource thereby garbage collecting resources of the test

• test_deploy_vm - our test scenario. All methods must begin with the test_ prefix

• VirtualMachine.creat( -> this is the deployVirtualMachineCmd)

• triple quote comments talk about the scenario in detail

• Multiple asserts are made with appropriate failure messages

• Read up about asserts here

• You can also use the should_dsl to do asserts as it is included with the Marvin install and is more readable

Running the test

IDE - Eclipse and PyDev

In PyDev you will have to setup the default test runner to be nose. For this:

• goto Window->Preferences->PyDev
• PyUnit - set the testrunner to nosetests
• In the options window specify the following

• --with-marvin
• --marvin-config=/path/to/demo.cfg
• --load

• Save/Apply

Now create a Debug Configuration with the project set the one in which you are writing your tests. And the main module to be your test_deploy_vm.py script we defined earlier. Hit Debug and you should see your test run within the Eclipse environment and report failures in the Debug Window. You will also be able to set breakpoints, inspect values, evaluate expressions while debugging like you do with Java code in Eclipse.

...

An astute reader would by now have found that the following pattern has been used in the test examples shown so far and in most of the suites in the test/integration directory:

• creation of an account

• deploying Vms, running some test scenario

• deletion of the account

This pattern is useful to contain the entire test into one atomic piece. It helps prevent tests from becoming entangled in each other ie we have failures localized to one account and that should not affect the other tests. Advanced examples in our basic verification suite are written using this pattern. Those writing tests are encouraged to follow the examples in test/integration/smoke directory.

...

Note that the testclient is available from the superclass using getClsTestClient in this case.

Checkin Tests

The agent simulator and marvin are integrated into maven build phases to help you run basic tests before pushing a commit. These tests are integration tests that will test the CloudStack system as a whole. Management Server will be running during the tests with the Simulator Agent responding to hypervisor commands. For running the checkin tests, your developer environment needs to have Marvin installed and working with the latest CloudStack APIs. These tests are lightweight and should ensure that your commit doesnt break critical functionality for others working with the master branch. The checkin-tests utilize marvin and a one-time installation of marvin will be done so as to fetch all the related dependencies. Further updates to marvin can be done by using the sync mechanism described later in this section.

These build steps are similar to the regular build, deploydb and run of the management server. Only some extra switches are required to run the tests and should be easy to recall and run anytime:

Building

Build with the -Dsimulator switch to enable simulator hypervisors

$ mvn -Pdeveloper -Dsimulator clean install

...

$ cd tools/marvin/marvin/sandbox/advanced
$ python advanced_env.py -i setup.properties -o advanced.cfgThese configurations are generated using the marvin configGenerator module. You can write your own configuration by following the examples shown in the configGenerator module:

• describe_setup_in_basic_mode()
• describe_setup_in_advanced_mode()
• describe_setup_in_eip_mode()

More detailed explanation of how the JSON configuration works is shown later sections of this tutorial.

...

Marvin can auto-generate these resource classes using API discovery. The auto-generation ability is being added as part of this refactor

Smoke

This is our so-called BVT - basic verification tests. Tests here include those that check the basic sanity of the cloudstack features. Include only simple tests for your feature here. If you are writing a check-in test, this the where the test module should be put.

Component

More in-depth tests drilling down the entire breadth of a feature can be found here. These are used for regression testing.

Devcloud Tests

Some tests have been tagged to run only for devcloud environment. In order to run these tests you can use the following command after you have setup your management server and the devcloud vm is running with tools/devcloud/devcloud.cfg as its deployment configuration.

...

The smoke tests and component tests contain attributes that can be used to filter the tests that you would like to run against your deployment. You would use nose attrib plugin for this. Following tags are available for filtering:

• advanced - Typical Advanced Zone

• basic - a basic zone without security groups

• sg - a basic zone with security groups

• eip - an elastic ip basic zone

• advancedns - advanced zone with a netscaler device

• advancedsg - advanced zone with security groups

• devcloud - tests that will run only for the basic zone on a devcloud setup done using tools/devcloud/devcloud.cfg

• speed = 0/1/2 (greater the value lesser the speed)

• multihost/multipods/mulitcluster (test requires multiple set of hosts/pods/clusters)

bash
$ nosetests --with-marvin --marvin-config=/path/to/config.cfg -w test_directory -a tags=advanced,tags=simulator # run tests tagged to run on an advanced zone with the simulator

Guidelines to choose scenarios for integration

There are a few do's and don'ts in choosing the automated scenario for an integration test. These are mostly for the system to blend well with the continuous test infrastructure and to keep environments pure and clean without affecting other tests.

Scenario

• Every test should happen within a CloudStack user account. The order in which you choose the type of account to test within should be
  User > DomainAdmin > Admin
  At the end of the test we delete this account so as to keep tests atomic and contained within a tenant's users space.

• Every test case should clean up it's resource ASAP, after it's no longer in use.
  • Any Create operation should be followed by adding the newly created object to the cleanup set. Otherwise any later failure before cleanup set updated would result in previous allocated object won't be freed.
  • Any resources allocated in setUp() for each test case must be released in tearDown(). Suggest using the variable "self.cleanup" to track the test case resources.
  • Any resources allocated in setUpClass() for whole test class must be released in tearDownClass(). Suggest using the variable "cls._cleanup" to track the class resources.
  • Any resources allocated (excepted in setUpClass()) must be added to the cleanup list then released in tearDown().

• All tests must be written with the perspective of the API. UI directions are often confusing and using the rich API often reveals further test scenarios. You can capture the API arguments using cloudmonkey/firebug.

• Tests should be generic enough to run in any environment/lab - under any hypervisor. If this is not possible then it is appropriate to mark the test with an @attr attribute to signify any specifics. eg: @attr(hypervisor='vmware') for runs only on vmware

• Every resource should be creatable in the test from scratch. Referring to an Ubuntu template is probably not a good idea. Your test must show how to fetch this template or give a static location from where the test can fetch it.

• Do not change global settings configurations in between a test. Make two separate tests for this. All tests run against one given deployment and altering the settings midway is not effective

Backend Verification with paramiko/and other means

• Verifying status of resources within hypervisors is fine. Most hypervisors provide standard SSH server access

• Your tests should include the complete command and its expected output. e.g. iptables -L INPUT to list the INPUT chain of iptables

• If you execute multiple commands then all of them should be chained together on one line e.g: service iptables stop; service iptables start #to stop and start iptables

• Your script must execute over ssh because this is how Marvin will execute it ssh <target-backend-machine> "<your script>"

• Most external devices like F5/ NetScaler have ssh open to execute commands. But you must include the command and its expected output in the test as not everyone is aware of the device's CLI

• If you are using a UI like vCenter to verify something most likely you cannot automate what you see there because there are no ASLv2 licensed libraries for vmware/esx as of today.

Python Resources

• The single largest python resource is the python website itself

• Mark Pilgrim's - 'Dive Into Python' - is another great resource. The book is available for free online. Chapter 1 to 6 cover a good portion of language basics and Chapter 13 & 14 are essential for anyone doing test script development.

• You can also use Zed Shaw's book, Learn Python the Hard Way, if programming is something completely new to you

• To read more about the assert methods the language reference is the ideal place - http://docs.python.org/library/unittest.html.

Acknowledgements

• The original author of the testing framework - Edison Su

• Maintenance and bug fixes - Prasanna Santhanam

• Documentation - Prasanna and Edison

For any feedback, typo corrections please email the dev@cloudstack.apache.org list.