Page History

Infrastructure, Simulator, Automation , new CI Changes for CloudStack:

Overview: With new modifications in CI, improvements in infrastructure automation like provisioning of hypervisor, auto install management server, enhancements in simulator, changes in test framework and test categorization, we can now run tests faster, in parallel and get CI with automated emailed notificationsImprovements in Infrastructure Automation, Simulator Enhancements, Marvin Framework enhancements,  Test Categorization, we can now automatically provision a Hypervisor Ex: XenServer, KVM etc, auto Install management server, automatically install Marvin, run parallel tests for a given build across multiple hypervisors, automatically log bugs, analyze logs collected, generate reports and email them with detailed information etc, are all possible.  With this, CI will automatically kick start every few hours, picking up latest code for a configured git branch and repo, does all the things mentioned above.

So, this is more of an end to end automation capability, and get CI run frequently with latest code and minimal effort.

With new self service changes, any body can request an automation run and get the report emailed to them easily. This will help them in validating their feature mergers and other easy validations.(Phase 2)

...

Automation Infrastructure Enhancements:

• Ability to install, run simulator and other hyper visors in different zones simultaneously. Run tests in parallel against multiple hypervisors, EX: Xenserver and KVM regression can now run in parallel,  provision easy automatic provisioning of kvm, xen and other hypervisors now available.

...

• Automatic bug filing system.

• Better utilization of test hardware: Simulator is run in a VM, real hardware is used only for provisioning tests.

...

Log Uploader Job: This job post the CI run uploads all logs to nfs share. All logs post the run are available under designated nfs share at nfs1at configured location in CI, EX: //example-nfs.ex.com/var/www/<4.4 version> etc. The logs uploaded here, will have test case run log, product logs etc available here for analysis for failures\run information. Once logs are uploaded, people can see logs in web UI and browse them from UI available at  by clicking on and selecting individual versions and hypervisors.EX: Http://10.147.38.151<Ip-address>/LogAnalyzer

BugLoggerJob: A job with bug logger service is created in jenkins, which will automatically log bugs to jira system, configured. This job currently can be run by QA post the run, by configuring the build, path of logs etc, and it will analyze logs uploaded as mentioned earlier and log bugs to jira system .

...

QA\Team Responsibilities in CI Run:  Once QA\Team identifies failures in the mail report sent above, and logs bugs for the failures, then they can mark individual test case with the logged bugid information. EX: If a test case by name “test_deploy_vm_start_failure” fails under test suite “test_deploy_vm.py” , the  it then  this test case need to be marked with bugid and push information as mentioned below. Push these changes as belowto repo. So, next time, when CI runs, this will disable test case will be disabled automatically from run in CI, till it gets fixed.

@attr(tags = ['selfservice'],BugId="234")

...

Developers Responsibilities in CI Run: Developers, once they have a bug raised for failures in CI, have to fix the bug, can and then remove the bugId information for from the respective test case. Once disabled, it the test case will again be part of CI run.

CI Run: In order to exclude failed test cases from running (for which there are bugs logged already, then while ) from re running again and again,  then while we are running CI, we have to add use below additional tags for bugId to exclude the test cases having bugslogged already from running as to nosetests as mentioned below.

nosetests --with-xunit --xunit-file=test_deploy_vm.xml --with-marvin --marvin-config=/home/santhosh/softwares/cs_4_4/cloudstack/setup/dev/advanced.cfg /home/santhosh/softwares/cs_4_4/cloudstack/test/integration/smoke/test_deploy_vm.py -a tags=selfserviceadvanced,'!BugId' --zone="Sandbox-simulator" --collect-only

...

https://cwiki.apache.org/confluence/display/CLOUDSTACK/Simulator+enhance mentsments

https://cwiki.apache.org/confluence/display/CLOUDSTACK/Writing+tests+leveraging+the+simulator+enhancements