Page History

...

1. Enable community to run QA tests in automated way
2. Make it easy to replicate a set-up
3. Enable to run BVTs continuously
4. Overall, work together as a community towards keeping master branch stable

It is important to note that keeping it simple and easy to replicate is one of the main goals of this exercise. Thus, it may not cater to every possible use case.
The code is based on Prasanna's effort on similar lines (big thanks to Prasanna!), and in future should enable the creation of test-beds for unified testing.

To ensure the master branch is relatively stable almost always, the proposal is to create a "staging" branch where all check-ins shall go. Once the BVTs pass against the staging branch, the commits may be merged with the master branch. Ideally, the commit process should be automated.

Test Run Requirements

1. Every test run should be isolated from previous test run
2. Clean-up hypervisors between runs
3. Spawn up new management server for each run

...

Typically, the setting up of the Cloud is a separate Jenkins job, upon successful run of which we launch a test suite in a different job. Note that, the code contains basic elements for you to set-up a single Jenkins job for both the steps, though it may be very slow (Since BVTs are run sequentially in this case to avoid Marvin issues)

Setting up from scratch

Follow the instructions here to set-up the infrastructure from scratch. This is a more involved task and needs familiarity with the CI tools.

It lists the set of commands / scripts used to set-up Cobbler, Puppet manifests etc. The sample configuration for DNSMASQ is provided and allow one to easily change to suit their needs.

Easy Set-Up

Setting up the infrastructure

1. Budget at least 4 machines to your set-up. Install XenServer (6+) on one machine to host Provisioner QACloud VM and management server. This machine is required to have internet access.
2. Set-up and configure IPMI network and enable hosts for IPMI. This link may be useful http://stuff.mit.edu/afs/athena/dept/cron/documentation/dell-server-admin/en/DRAC_5/racugc1j.htm
3. Set-up private switch for hosts on eth0. Reference config is available here
4. Import QACloud template by downloading from here
5. Change MAC address in Cobbler (cobbler system edit --mac-address=<MAC> --name=xen1/2/3 ), DNSMASQ (/etc/dnsmasq.conf) and system.properties file
6. Set IPMI password in configure script
7. Set-up public internet access for Provisioner QACloud VM so it may proxy requests as needed. This may involve setting up correct ip routes. (Be careful here, as incorrect configs for routes / DNS will manifest in weird network boot issues)

Advanced Options

1.  If the switch config does not  mimic provided reference, please change the IP address for DNSMASQ and system.properties file
2. Set correct routes for IP configuration in Provisioner QACloud and its host XenServer

The configuration would look something like the below after things are correctly set-up, after which you should proceed to adding Jenkins jobs as described in next section.

...

python2.7 /root/cloud-autodeploy/cherryPickCommits.py <JenkinsUrl> <JenkinsUser> <JenkinsPassword>
python2.7 /root/cloud-autodeploy/kickContinuousBuild.py <JenkinsUrl> <JenkinsUser> <JenkinsPassword> <Jenkins Job Name>

...

Merging commits from staging to master

...

This part relates to the overall goal of keeping the master branch relatively stable, and make cherry-picking of commits an easier process rather than someone monitoring a mailing list.

Thus, all BVTs run against a "staging" branch. It is important to note that "master" may not just be the repository master branch, but a term used to refer to any branch you may want to keep stable so as to create a RC out of.

The flow here is :

1. Kick off a Python script after BVT run against "staging"
2. If the BVT run is successful, get the commit SHAs from last successful BVT run to current run and merge the commits to "master"
   1. In case or merge / git  rebase failure, notify developers.
3. If BVT run is unsuccessful, get the commit SHAs from last successful BVT run to current run, and mail developers to look at results and see if their commit broke the BVT

Thus, the flow ensures that only clean commits make its way to master branch. This should enable us to go one step further in making master almost always stable.

The obvious question to ask is : how do we ensure commits pass successfully across all hypervisor configuration types?

This goes back to providing a unified design to set-up infrastructure, and tie all configurations together using Jenkins job. At the end of the Jenkins job, the above flow shall be kicked off.

Troubleshooting

• Cobbler / DNSMASQ issues : Useful to check ip routes in this case. These will manifest as PXE boot errors or DHCP sending a hostname instead of IP address in PXE response. This has been a source of lot of frustration.
• Puppet errors : Check certificates. Remove .pem file in /var/lib/puppet/ssl if empty
• Marvin issues : Python "requests" package may mask many errors that may cause Marvin to fail when seemingly things are right from Management Server logs. Check errors from this package when see things failing randomly.
• Management Server issues : Very rarely the management server fails to come up even after service cloudstack-management restart scripts returns fine. Currently know of no good resolution to this but to wait till next deployment is done

Enhancements

1. Integrate across multiple hypervisor runs

2. Better scheduling using a pool of hosts

4. Providing hooks so one can customize actions to be taken in case of failures.

Open Questions

1. The VM is pretty big. Where do we host it? People can set up using the provided instructions too

2. Have a Hackathon to improve this / set up from scratch?

3. To keep template size manageable, the VM has ~50GB of space for NFS. This may run out quickly. Add instructions to add a new volume to VM to serve as NFS?