Page History

Table of Contents

Introduction

Marvin - our automation framework is a Python module that leverages the abilities of python and its multitude of libraries. Tests written using our framework use the unittest module under the hood. The unittest module is the Python version of the unit-testing framework originally developed by Kent Beck et al and will be familiar to the Java people in the form of JUnit. The following document will act as a tutorial introduction to those interested in testing CloudStack with python.

...

~/workspace/cloudstack/incubator-cloudstack(branch:master*) » nosetests --with-marvin --marvin-config=tools/devcloud/devcloud.cfg --load -a tags='devcloud' test/integration/smoke

Test Deploy Virtual Machine ... ok
Test Stop Virtual Machine ... ok
Test Start Virtual Machine ... ok
Test Reboot Virtual Machine ... ok
Test destroy Virtual Machine ... ok
Test recover Virtual Machine ... ok
Test destroy(expunge) Virtual Machine ... ok

----------------------------------------------------------------------

Ran 7 tests in 0.001s

OK

Guidelines to choose scenarios for integration

There are a few do's and don'ts in choosing the automated scenario for an integration test. These are mostly for the system to blend well with the continuous test infrastructure and to keep environments pure and clean without affecting other tests.

Scenario

• Every test should happen within a CloudStack test account. The order in which you choose the type of account to test within should be:

                                   User > DomainAdmin > Admin

           At the end of the test we delete this account so as to keep tests atomic and contained within a tenant's users space.

• All tests must be written with the perspective of the API. UI directions are often confusing and using the rich API often reveals further test scenarios. You can capture the API arguments using cloudmonkey/firebug.
• Tests should be generic enough to run in any environment/lab - under any hypervisor.
  • If this is not possible then it is appropriate to mark the test with an @attr attribute to signify any specifics. eg: @attr(hypervisor='vmware') for runs only on vmware
• Every resource should be creatable in the test from scratch.
  • referring to an Ubuntu template is probably not a good idea. Your test must show how to fetch this template or give a static location from where the test can fetch it.
• Do not change global settings configurations in between a test. Make two separate tests for this. All tests run against one given deployment and altering the settings midway is not effective

Backend Verification with paramiko/and other means

• Verifying status of resources within hypervisors is fine. Most hypervisors provide standard SSH server access
• Your tests should include the complete command and its expected output. 
  • e.g. iptables -L INPUT -# to list the INPUT chain of iptables
• If you execute multiple commands then all of them should be chained together on one line
  • e.g: service iptables stop; service iptables start #to stop and start iptables
• Your script must execute over ssh because this is how Marvin will execute it
  • ssh <target-backend-machine> "<your script>" #should return the output of your script
  • move the credential specific information into the deployment config file and/or use a standard credential
• Most external devices like F5/ NetScaler have ssh open to execute commands. But you must include the command and its expected output in the test as not everyone is aware of the device's CLI
• If you are using a UI like vCenter to verify something most likely you cannot automate what you see there because there are no ASLv2 licensed libraries for vmware/esx as of today.

Python Resources

1. The single largest python resource is the python website itself - http://www.python.org
2. Mark Pilgrim's - "Dive Into Python" - is another great resource. The book is available for free online - http://www.diveintopython.net. Chapter 1- 6 cover a good portion of language basics and Chapter 13 & 14 are essential for anyone doing test script development
3. To read more about the assert methods the language reference is the ideal place - http://docs.python.org/library/unittest.html.

...