Page History

...

• Smoke Test: This is run against devCloud and CloudStack simulator.  It allows the developer to verify that their code changes did not break the business logic in CloudStack's self-provisioning code.  This test is to be run by all developers before they check into a release branch or master.
• Build Verification Test: This is a continuous integration test that runs on every checkin to verify that the code did not break basic functionality in multiple platforms.  The tests here tests actual provisioning on physical and virtual resources.  Any code that breaks the BVT must be reverted.
• Regression Test: This is the complete functionality test on CloudStack.  This may not run on a continuous basis as it may take a long time to run.  The timing on when this is run is left up to the release manager of that releasetests CloudStack's business logic and is run against the simulator only.  Code that causes breakage in Regression Tests must be treated as P0 bugs.  If bugs are not fixed within 24 hours, the code that causes it must be reverted. 
• Failure Injection Tests: This is a set of tests to simulate failures in the CloudStack system.  These tests do not have to run continuously but must be run before every release.
• Scalability Tests: This set of tests utilizies the CloudStack simulator to simulate tens of thousands of hosts and deploying hundreds of thousands of VMs.

...

• BVT and Regression Tests must be fully automated.  This means it is automated in all phases: deployment, setup, execution, reporting, and bug filing.
• BVT must be run continuously.
• Regression Tests are run at an interval decided upon by QA.  It is recommended that it runs daily.must be run continuously.
• Any code that breaks BVT and Regression Tests on master and all release branches must be reverted.
• All tests are maintained by developers and QA.  If  All tests must pass the BVT and Regression Tests before it is run as part of the normal BVT.  Once a test is checked in, if a feature changes CloudStack behavior to cause a breakage in the tests, it is the feature developer's responsibility to fix the tests.
• All tests are driven using API.  No UI testing.

...

• Before pushing changes, every developer must run the smoke tests against their machine.  You should add tests to the smoke tests to test your own feature if possible. Smoke test requirement is to be completed under 15 minutes from begin to finish.  If your functionality cannot be tested on devCloud, then you can write or add to the existing simulator to simulate responses for your feature.
• For big changes, a feature branch must be broken off.  The developers can request BVT and regression tests to be run against their feature before merging into master/release branch.  Once requested, the automation test system will schedule a feature branch to be tested and results will be provided.  
• All checkins/merges to master/release branch is done on a staging branch.  Only when the checkin passed BVT and regression tests are they cherry-picked into the master/release branch.
• A feature branch cannot merge into master/release with any BVT and regression test failures.
• Failure Injection Tests and Scalability Tests must be run before a release is completed.

...

• Runs continuously on BVT hardware.
• Tests the common deployments on common hardware.  For now, the common hardware are KVM, XenServer, VmWare.
• Community proposes and decides on tests that goes into BVT.
• Must be completed from deploy to finish within one hour.

Regression Test

• Runs at a set interval decided on by the release manager.  Recommendation is to run once a daycontinuously on simulator.
• Full functionality testing.No set limit on time to completion
• Every production bug must have a regression test design for it or explanation why one is not necessary.

...