Page History

This page specifies the rules and guidelines for developers and QA when dealing with CloudStack 's BVT and automated regression testsAutomated Tests.

Why

...

Automation Tests

CloudStack deals with a wide range of hardware resources physical and virtual resources.  It is impossible for anyone any one developer/QA to test if their feature/functionality works across all of the supported resources.  CloudStack's BVT and automated tests CloudStack Automated Tests provide a method for developers to submit their changes to be tested across the supported infrastructure.  It also allows support for 

How to donate hardware/software for integration testing

Rules and Guidelines for Developers and QA

...

.  Physical/Virtual resource providers can also donate their resources and tests to the Automated Test setup to have their resources be tested as part of every release.

Types of Automated Tests

CloudStack Automated Tests consist of the following setups for used in different situations.

• Smoke Test: This is run against devCloud and CloudStack simulator.  It allows the developer to verify that their code changes did not break the business logic in CloudStack's self-provisioning code.  This test is to be run by all developers before they check into a release branch or master.
• Build Verification Test: This is a continuous integration test that runs on every checkin to verify that the code did not break basic functionality in multiple platforms.  The tests here tests actual provisioning on physical and virtual resources.  Any code that breaks the BVT must be reverted.
• Regression Test: This is the complete functionality test on CloudStack.  This tests CloudStack's business logic and is run against the simulator only.  Code that causes breakage in Regression Tests must be treated as P0 bugs.  If bugs are not fixed within 24 hours, the code that causes it must be reverted. 
• Failure Injection Tests: This is a set of tests to simulate failures in the CloudStack system.  These tests do not have to run continuously but must be run before every release.
• Scalability Tests: This set of tests utilizies the CloudStack simulator to simulate tens of thousands of hosts and deploying hundreds of thousands of VMs.

Design Principles

• BVT and Regression Tests must be fully automated.  This means it is automated in all phases: deployment, setup, execution, reporting, and bug filing.
• BVT must be run continuously.
• Regression Tests must be run continuously.
• Any code that breaks BVT and Regression Tests on master and all release branches must be reverted.
• All tests are maintained by developers and QA.  All tests must pass the BVT and Regression Tests before it is run as part of the normal BVT.  Once a test is checked in, if a feature changes CloudStack behavior to cause a breakage in the tests, it is the feature developer's responsibility to fix the tests.
• All tests are driven using API.  No UI testing.

Rules and Guidelines for Developers and QA

• Before pushing changes, every developer must run the smoke tests against their machine.  You should add tests to the smoke tests to test your own feature if possible. Smoke test requirement is to be completed under 15 minutes from begin to finish.  If your functionality cannot be tested on devCloud, then you can write or add to the existing simulator to simulate responses for your feature.
• For big changes, a feature branch must be broken off.  The developers can request BVT and regression tests to be run against their feature before merging into master/release branch.  Once requested, the automation test system will schedule a feature branch to be tested and results will be provided.  
• All checkins/merges to master/release branch is done on a staging branch.  Only when the checkin passed BVT and regression tests are they cherry-picked into the master/release branch.
• A feature branch cannot merge into master/release with any BVT and regression test failures.
• Failure Injection Tests and Scalability Tests must be run before a release is completed.

Test Properties

...

Smoke Test

...

• Maintain by developers
• Runs against simulator and devCloud
• Checks mostly self-provisioning logic and error/success handling of simulated resource responses
• Must be completed from deploy to finish within 15 minutes
• Tests written for smoke test can be taken by QA to expanded on to be 

...

Build Verification Test (BVT)

...

• Runs continuously on BVT hardware.
• Tests the common deployments on common hardware.  For now, the common hardware are KVM, XenServer, VmWare.
• Community proposes and decides on tests that goes into BVT.
• Must be completed from deploy to finish within one hour.

...

Regression Test

...

• Runs continuously on simulator.
• Full functionality testing.
• Every production bug must have a regression test design for it or explanation why one is not necessary.

...

Fault Injection Test

...

• Introduces external faults such as network failure, host failure, storage failure simulation through the simulator.
• Tests CloudStack's behavior in response to such failures.
• Maintain by developers because only they can really know what their system's behavior is on failure.
• Must be run before every release.
• Production fault handling bugs must have a test written in this test before the bug is closed.

...

Scalability Test

...

• Tests the scalability of CloudStack's services, including management server, console proxy, and ssvm.
• Uses the simulator to the scalability.
• All results and test conditions must be recorded on the scalability wiki.

How to donate hardware/software for integration testing