Page History

...

1. Enable community to run QA tests in automated way
2. Make it easy to replicate a set-up
3. Enable to run BVTs continuously
4. Overall, work together as a community towards keeping master branch stable

It is important to note that keeping it simple and easy to replicate is one of the main goals of this exercise. Thus, it may not cater to every possible use case.
The code is based on Prasanna's effort on similar lines (big thanks to Prasanna!), and in future should enable the creation of test-beds for unified testing.

To ensure the master branch is relatively stable almost always, the proposal is to create a "staging" branch where all check-ins shall go. Once the BVTs pass against the staging branch, the commits may be merged with the master branch. Ideally, the commit process should be automated.

Requirements

1. Every test run should be isolated from previous test run
2. Clean-up hypervisors between runs
3. Spawn up new management server for each run

...

python2.7 /root/cloud-autodeploy/cherryPickCommits.py <JenkinsUrl> <JenkinsUser> <JenkinsPassword>
python2.7 /root/cloud-autodeploy/kickContinuousBuild.py <JenkinsUrl> <JenkinsUser> <JenkinsPassword> <Jenkins Job Name>

...

Merging commits from staging to master

...

This part relates to the overall goal of keeping the master branch relatively stable, and make cherry-picking of commits an easier process rather than someone monitoring a mailing list.

Thus, all BVTs run against a "staging" branch. It is important to note that "master" may not just be the repository master branch, but a term used to refer to any branch you may want to keep stable so as to create a RC out of.

The flow here is :

1. Kick off a Python script after BVT run against "staging"
2. If the BVT run is successful, get the commit SHAs from last successful BVT run to current run and merge the commits to "master"
   1. In case or merge / git  rebase failure, notify developers.
3. If BVT run is unsuccessful, get the commit SHAs from last successful BVT run to current run, and mail developers to look at results and see if their commit broke the BVT

Thus, the flow ensures that only clean commits make its way to master branch. This should enable us to go one step further in making master almost always stable.

The obvious question to ask is : how do we ensure commits pass successfully across all hypervisor configuration types?

This goes back to providing a unified design to set-up infrastructure, and tie all configurations together using Jenkins job. At the end of the Jenkins job, the above flow shall be kicked off.

Troubleshooting

• Cobbler / DNSMASQ issues : Useful to check ip routes in this case. These will manifest as PXE boot errors or DHCP sending a hostname instead of IP address in PXE response. This has been a source of lot of frustration.
• Puppet errors : Check certificates. Remove .pem file in /var/lib/puppet/ssl if empty
• Marvin issues : Python "requests" package may mask many errors that may cause Marvin to fail when seemingly things are right from Management Server logs. Check errors from this package when see things failing randomly.
• Management Server issues : Very rarely the management server fails to come up even after service cloudstack-management restart scripts returns fine. Currently know of no good resolution to this but to wait till next deployment is done

...