Page History

This page specifies the rules and guidelines for developers and QA when dealing with CloudStack 's BVT and automated regression testsAutomated Tests.

Why

...

Automation Tests

CloudStack deals with a wide range of physical and virtual resources.  It is impossible for any one developer/QA to test their feature/functionality across all of the supported resources. CloudStack 's BVT and regression tests Automated Tests provide a method for developers to submit their changes to be tested across the supported infrastructure.  Physical/Virtual resource providers can also donate their resources and tests to the Automated Test setup to have their resources be tested as part of every release.

Types of Automated Tests

CloudStack Automated Tests consist of the following setups for used in different situations.

• Smoke Test: This is run against devCloud and CloudStack simulator.  It allows the developer to verify that their code changes did not break the

...

• basic business logic in CloudStack's self-provisioning code.  This test is to be run by all developers before they check into a release branch or master.
• Build Verification Test: This is a continuous test that runs on every checkin to verify that the code did not break basic functionality in multiple platforms.  Any code that breaks the BVT must be reverted.
• Regression Test: This is the complete functionality test on CloudStack.  This may not run on a continuous basis as it may take a long time to run.  The timing on when this is run is left up to the release manager of that release.  Code that causes breakage in Regression Tests must be treated as P0 bugs.  If bugs are not fixed within 24 hours, the code that causes it must be reverted. 
• Failure Inject Tests: This is a set of tests to simulate failures in the CloudStack system.  These tests must be run before 

Principles

• BVT and Regression Tests must be fully automated.  This means it is automated in all phases: deployment, setup, execution, reporting, and bug filing.
• Any code that breaks BVT and Regression Tests on master and all release branches must be reverted.
• Any code that breaks BVT and Regression Tests on feature branches will be marked as P0 bug for that feature.
• All tests are maintained by developers and QA.  If a feature changes CloudStack behavior to cause a breakage in the tests, it is the feature developer/s responsibility to fix the tests 

How to donate hardware/software for integration testing

Rules and Guidelines for Developers and QA

Principles behind