Page History

...

The central jenkins (master) instance at jenkins.bacd.org connects to a jenkins slave within the data center via SSH. Traditionally a non-standard port with key-pair only access is preferred to keep the deployment secure. This slave is a VM with necessary packages, scripts and tools that drives the rest of the dedicated test infrastructure through various configurations required for testing. All the necessary configurations are controlled by a matrix of combinations. These combinations are nothing but jobs on the jenkins (master).

...

• cobbler - all baremetal PXE booting from ISO images. Preloaded (RHEL, Ubuntu, XenServer, ESXi) images
• puppetmaster - puppet is used for the configuration of packages within the freshly deployed management server VM.
• dnsmasq - Manages all the DNS for the dedicated testbed.
• python and virtualenv - tests run within the virtualenv environment in each run
• httpd and yumrepo - an ephemeral yum repository holding latest packages from a development branch

...

Key components

1. At the center of the workflow is the "jenkins slave" appliance that manages the infrastructure. This is a Cent OS 6.2 VM running on a XenServer. The slave VM is responsible for triggering the entire test run when it is time. This schedule is indicated by jenkins master through the test jobs.

...

ipmi for power management is setup on all the test servers and the ipmitool provides a convienient CLI for booting the machines on the network into PXEing.

marvin integration

Once CloudStack has been installed and the hypervisors prepared we are ready to use marvin to stitch together zones, pods, clusters and compute and storage to put together a 'cloud'. Once configured - we perform a cursory health check to see if we have all systemVMs running in all zones and that built-in templates are downloaded in all zones. Subsequently we are able to launch tests on this environment

Only the latest tests from git are run on the setup. This allows us to test in a pseudo-continuous fashion with a nightly build deployed on the environment. Each test run takes a few hours to finish.

Control via github

There are two github repositories controlling the test infrastructure.
a. The Puppet recipes at gh:acs-infra-test
b. The gh:cloud-autodeploy repo that has the scripts to orchestrate the overall workflow

Workflow

When jenkins triggers the test job following sequence of actions occur on the test infrastructure

1. The deployment configuration is chosen based on the hypervisor being used. We currently have xen.cfg and kvm.cfg that are in the gh:cloud-autodeploy repo
2. A virtualenv python environment is created within which the configuration and test runs by marvin are isolated into. Virtualenv is great for sandboxing test environment runs. In to the virtualenv are copied all the latest tests from the cloudstack repo.
3. We fetch the last successful marvin build and install it within this virtualenv. Installing a new marvin on each run helps us test with the latest APIs available.
4. We fetch the latest version of the driver script from github:cloud-autodeploy. fetching the latest allows us to make adjustments to the infra without having to copy scripts in to the test infrastructure.
5. Based on the hypervisor chosen we choose a profile for cobbler to reimage the hosts in the infrastructure. If xen is chosen we bring up the profile of the latest xen kickstart available in cobbler. currently - this is at xenserver 6.2. If KVM is chosen we can pick between ubuntu and rhel based host OS kickstarts. Other hypervisors such as ESXi and XCP are also bootable.
6. With this setup we kick off the driver script with the following cmd line arguments

   $ python configure.py -v $hypervisor -d $distro -p $profile -l $LOG_LVL

   The $distro argument chooses the hostOS of the mgmt server - this can be ubuntu / rhel. LOG_LVL can be set to INFO/DEBUG/WARN for troubleshooting and more verbose log output.

7. The configure script does various operations to prepare the environment:
   • clears up any dirty cobbler systems from previous runs
   • cleans up Puppet certificates of these systems. Puppet recipes will fail if puppetmaster finds an invalid certificate
   • starts up a new xenserver VM that will act as the mgmt server. we chose to keep things simple by launching the vm on a xenserver. one could employ jclouds via jenkins to deploy the mgmt server VM on a dogfooded cloudstack.
   • in parallel the deployment config of marvin is parsed through to find the hypervisors that need to be cleaned up, pxe booted and prepared for the CloudStack deployment.
   • all the hosts in the marvin config are pxe booted via ipmi and cobbler takes over to reimage them with the profile chosen by the jenkins job run.
   • while this is happening we also seed the secondary storage with the systemvm template reqd for the hypervisor.
   • all the primary stores in the marvin config are then cleaned for the next run.
8. While cobbler is reimaging the hosts with the right profiles, the configure script waits until all hosts are reachable over ssh. It also checks for essential services (http, mysql) ports to come up. Cobbler once done with refreshing the machine hands over the reins to puppet.
9. Puppet slaves within the machines in the environment reach out to puppetmaster to get their identity. mgmt server vm fetches its own recipe and starts configuring itself while hypervisors will do the same in case they need to be acting as KVM agents.
10. When the essential ports for mgmt server - 8080 and 8096 are open and listening we know that the mgmt server has come up successfully. We then go ahead and deploy the configuration specified by marvin.
11. After marvin finishes configuring the cloud - it performs a health check to see if the system is ready for running tests upon.
12. Tests are run using the nose test runner with the marvin plugin and reports are recorded by jenkins.

Jobs in the jenkins test pipeline

The jenkins pipeline (sequence of jobs) for the above workflow of tests is described below in flowchart format.

View the configuration of test-matrix/test-matrix-extended jobs to understand further.

• test-matrix-* - This is the umbrella job that starts all other jobs in the sequence. This job points to master by default and so does test-yumrepo-refresh. When you want to change the branch you have to change it for both jobs to ensure your tests are running on the right target branch.
• test-yumrepo-refresh - hourly job refreshing packages on a s3 bucket
• test-environment-refresh - cleanup all hosts to a target cobbler profile (xen/xcp/kvm)
• test-packaging - deploy new cloudstack vm, configure packages, install cloudstack latest rpms from s3
• test-setup-advanced-zone - setup an advanced zone on the testbed and wait for systemvms to be ready
• test-smoke|regression-matrix - run the bvt/full regression tests on the environment
• test-maintenance-operations - run any tests that need to run sequentially like maintenance mode on hosts, disabling zone etc.

Limitations

1. Currently this infrastructure has limited capacity and with the increasing number of tests we sometimes run out of resources during a test run
2. Only advanced zone and basic zone with security groups are currently employed. Any external networking device etc is not employed in this testbed
3. No object storage or ceph storage

Enhancements

• run test fixes on idle environment upon checkin without deploy
• creating ones own slave vm appliance
• dogfooding the slave appliance
• run test fixes on idle environment upon checkin without deploy
• via packer
• custom zones - using a marvin config file
• logging enhancements = archiving + syslog
• digest emails via jenkins. controlling spam
• external devices (LB, VPX, FW)

Future

1. monitoring and status page for the infra
2. control all jenkins jobs via github using jenkins-job-builder
3. multiple internal jenkins environments on a single landing page jenkins.bacd.orgnot everyone deploys CloudStack the same
4. multiple hv environments with multiple hv configurations
5. multiple storage configurations - ceph, object store etc

Troubleshooting

Any of the several steps in the workflow can fail. If you find a failure, please bring the issue to notice on the dev@ mailing list.