Infrastructure, Simulator, Automation , new CI Changes for CloudStack:
overview: With modifications, improvements in infrastructure automation like provisioning of hyper visor, install management server, enhancements in simulator, changes in test framework and test categorization, we can now run tests faster, in parallel and get CI with automated emailed notifications.
With new self service changes, any body can request an automation run and get the report emailed to them easily. This will help them in validating their feature mergers and other easy validations.
High level Objectives of these changes:
Automate the semi-manual testing procedure that is being followed now.
Automate log gathering and bug filing.
Optimize the test runs so that they can be used as self service and can be run with out hardware, for continuous integration and validation of business logic.
Fix the simulator so that it is in sync with the new features added during last few releases
Add capability to the simulator for fault injection tests,timeouts etc.
Self service regression tests that community can use to verify their changes.
Provide Profiling and coverage Information.
Simulator now up to date with current features.
Test cases are now categorized as simulator based(selfservice) and hardware dependent(provisioning).
Simulator based tests can be used for regression check by developers in their developers setup itself with out real hardware.
Reduced test cycle times by enabling parallel test execution in multiple zones. For example simulator and provisioning test cases can run in different zones parallel.
Improved resource management, tests can be parallel executed and queued.
About Simulator :
Run tests without using actual hardware, eliminates setup and reduces test execution time.
With bug fixes and enhancements, additional tests can now run against simulator and these can be automated for better
With the enhancements additional tests can now run against simulator resulting in improved code coverage
Simulator: Comparison (old vs new) :
Old Simulator | New Simulator Improvements |
Mocks only success scenarios | Supports simulating faults-injections, timeouts |
No way to change/cleanup mock | Allows defining test case specific mocks which can defined and so applicable for entire test be cleaned up as appropriate suite |
Test only basic scenarios | Ability to test complex scenarios like VM. deployment retry logic, HA, VM migration etc. |
No way to change/cleanup mock once.No way to check if the mocked behavior is executed. Limited code coverage | Ability to check if mock executed. This makes tests more deterministic . With the ability to tests failure/negative scenarios. Tests additional tests can be quickly developed thereby improving overall coverage. Enables devs to do better upfront testing rather then relying on QA team, should result in less bugs late in the cycle |
Validating check-ins for your local changes, using Simulator
• From developers' point of view, whenever they do a change on their local environment, they can now easily validate the changes by running “selfservice" tests before check-in the change.
• Using "simulator", they can run these tests locally on their development environment, without requiring any hardware . This will help them to make sure that CS code is not broken with their changes, by running basic integration tests.
Simulator: Next steps
• Think of simulator as a first class HV, feature specification should clearly call it out in the list of supported HVs. Feature devs needs to keep the simulator up to date with respective changes.
• Devs/QA to write new tests for both existing and new features (target should be to improve test coverage of the overall product code)
Automation Infrastructure Enhancements:
• Ability to run simulator and other hyper visors in different zones simultaneously. Xenserver and KVM regression can now run in parallel, provision easy kvm,xen and other hypervisors now available.
• Automatic collection of logs. Addition of analysis engine for log analysis post the run .
• Automatic bug filing system
• Better utilization of test hardware: Simulator is run in a VM, real hardware is used only for provisioning tests.
• Queuing of jobs if capacity of resources is reached.
• Use of oss and easy to use tools like cobbler and puppet with extensible jenkins system eases addition of new
New CI Run Design and WorkFlow: <will be added soon>
Email Notification and Reporting: With CI runs continuously for a given version of CS, reports will be automatically emailed with below information.
1. Test Run Information: The report contains test run and other meta data information.
EX:
Build URL | http://jenkins.abcd.com/job/report_generator_47_Sandbox-simulator_simulator.xml/5/ |
Project: | report_generator_47_Sandbox-simulator_simulator.xml |
Build Start Date: | Wed May 14 16:55:57 EST 2014 |
Build End Date: | Wed May 14 16:55:57 EST 2014 |
Git Repo Url: | |
Git Commit Id: | 260e06d64c07c6e5f3c133d8bdc2779fad62c672 |
Git Branch Info: | 4.4 |
Zone Name: | xs |
Hyper Visor Info: | Simulator |
Build No: | 47 |
Build Detailed Report: | http://jenkins.abcd.com/job/${ENV,var="JOB_NAME"}/5/testReport/ |
Wiki Links: | Http://cwiki.apache.org |
Build Cause: | Started by user |
Build Description: | |
Built on: | cobbler-SC |
2. Test case Success,Error,Skipped count.
3. Individual test case pass\fail description. If there are bugs logged already for a test case, then it will be mentioned as a known issue.
Log Uploader Job: This job post the CI run uploads all logs to nfs share. All logs post the run are available under designated nfs share at nfs1.ex.com/var/www/<4.4 version> etc. The logs uploaded here, will have test case run log, product logs etc available here for analysis for failures\run information. Once logs are uploaded, people can see logs in web UI and browse them from UI available at by clicking on and selecting individual versions and hypervisors.EX: Http://10.147.38.151/LogAnalyzer
BugLoggerJob: A job with bug logger service is created in jenkins, which will automatically log bugs to jira system, configured. This job currently can be run by QA post the run, by configuring the build, path of logs etc, and it will analyze logs uploaded as mentioned earlier and log bugs to jira system .
QA\Team Responsibilities in CI Run: Once QA\Team identifies failures in the mail report sent above, and logs bugs for the failures, then they can mark individual test case with bugid information. EX: If a test case by name “test_deploy_vm_start_failure” fails under test suite “test_deploy_vm.py” , the it need to be marked with bugid and push these changes as below. So, this will disable test case from run in CI, till it gets fixed.
@attr(tags = ['selfservice'],BugId="234")
def test_deploy_vm_start_failure(self):
Developers Responsibilities in CI Run: Developers once they fix the bug, can remove the bugId information for the test case. Once disabled, it will again be part of CI run.
CI Run: In order to exclude test cases from running for which there are bugs logged already, then while running CI, we have to add tags for bugId to exclude the test cases having bugs logged already from running as below.
nosetests --with-xunit --xunit-file=test_deploy_vm.xml --with-marvin --marvin-config=/home/santhosh/softwares/cs_4_4/cloudstack/setup/dev/advanced.cfg /home/santhosh/softwares/cs_4_4/cloudstack/test/integration/smoke/test_deploy_vm.py -a tags=selfservice,'!BugId' --zone="Sandbox-simulator" --collect-only
Code Changes Location:
1. Infrastructure Code: All the Infrastructure code related to the new CI changes are maintained at the private repo. Once decided and cleaned, we will upload those changes to a particular location either inside of CS or otherwise.
2. Marvin\TestCode Changes: All the marvin\test code changes are available as part of CS git Repository and relevant branch itself.
https://git-wip-us.apache.org/repos/asf/cloudstack.git
Links\ References:
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Simulator+enhance ments