Page History

...

This document talks about the continuously evolving continuous test infrastructure used to setup, deploy, configure, and test Apache CloudStack. Information here is useful for anyone involved in build, test, continuous integration even operators of CloudStack.

...

a. Cobbler - cobbler is a provisioning PXE server (and much more) useful for rapid setup of linux Linux machines. It can do DNS, DHCP, power management and package configuration via puppet. It is capable of managing network installation of both physical and virtual infrastructure. Cobbler comes with an expressive CLI as well as web-ui frontends for management.

...

profiles - these are text files called kickstarts defined for a distribution's installation. For eg: RHEL 6.1 or Ubuntu 12.04 LTS. Each of the machines in the test environment - hypervisors and cloudstack CloudStack management servers contains a profile in the form of a kickstart.

...

root@infra ~# cobbler profile list
cloudstack-rhel
cloudstack-ubuntu
rhel63-kvmKVM
rhel63-x86_64
ubuntu1204-x86_64
xen602
xen56

...

b. Puppet master - Cobbler reimages machines on-demand but it is upto puppet Puppet recipes to do configuration management within them. The configuration management is required for kvm KVM hypervisors (kvm KVM agent for eg and for the cloudstack CloudStack management server which needs mysql, cloudstack, etc. The puppetmasterd daemon on the driver-vm is responsible for 'kicking' nodes to initiate configuration management on themselves when they come alive.

So the driver-vm is also the repository of all the puppet Puppet recipes for various modules that need to be configured for the test infrastructure to work. The modules are placed in /etc/puppet Puppet and bear the same structure as our github repo. When we need to affect a configuration change on any of our systems we only change the github repo and the systems in place are affected upon next run.

c. dnsmasq - DNS is controlled by cobbler but its configuration of hosts is set within dnsmasq.d/hosts. This is a simple 1-1 mapping of hostnames with IPs. For the most part this should be the single place where one needs to alter for replicating the test setup. Everywhere else only DNS names are/should-be used.

d. dhcp DHCP - DHCP is also done by dnsmasq. All configuration is in /etc/dnsmasq.conf. static mac-ip-name mappings are given for hypervisors while the virtual instances get dynamic ips

e. ipmitool - ipmi for power management is setup on all the test servers and the ipmitool provides a convienient cli CLI for booting the machines on the network into PXEing.

...

3. Hypervisor hosts - There currently are 4 hosts in this environment. These are arranged at the moment in three pods so as to be capable of being deployed in a two zone environment. One zone with two pods and and a second zone with a single pod. This covers tests that depend on
a. single zone/pod/cluster
b. multiple cluster
c. inter-zone tests
d. multi-pod tests

marvin integration

once cloudstack CloudStack has been installed and the hypervisors prepared we are ready to use marvin to stitch together zones, pods, clusters and compute and storage to put together a 'cloud'. once configured - we perform a cursory health check to see if we have all systemVMs running in all zones and that built-in templates are downloaded in all zones. Subsequently we are able to launch tests on this environment

...

there are two github repositories controlling the test infrastructure.
a. The puppet Puppet recipes at gh:acs-infra-test
b. The gh:cloud-autodeploy repo that has the scripts to orchestrate the overall workflow

...

1. The deployment configuration is chosen based on the hypervisor being used. We currently have xen.cfg and kvmKVM.cfg that are in the gh:cloud-autodeploy repo

2. A virtualenv python environment is chosen within which the configuration and test runs by marvin are isolated into. Virtualenv is great for sandboxing test environment runs. In to the virtualenv are copied all the latest tests from the git:incubator-cloudstack CloudStack repo.

3. we fetch the last successful marvin build from builds.a.o and install it within this virtualenv. installing a new marvin on each run helps us test with the latest APIs available.

...

5. based on the hypervisor chosen we choose a profile for cobbler to reimage the hosts in the infrastructure. if xen is chosen we bring up the profile of the latest xen kickstart available in cobbler. currently - this is at xen 6.0.2. if kvm KVM is chosen we can pick between ubuntu and rhel based host OS kickstarts.

...

7. The configure script does various operations to prepare the environment:
a. clears up any dirty cobbler systems from previous runs
b. cleans up puppet Puppet certificates of these systems. puppet Puppet recipes will fail if puppetmaster finds an invalid certificate
c. starts up a new xenserver VM that will act as the mgmt server. we chose to keep things simple by launching the vm on a xenserver. one could employ jclouds via jenkins to deploy the mgmt server VM on a dogfooded cloudstack.
d. in parallel the deployment config of marvin is parsed through to find the hypervisors that need to be cleaned up, pxe booted and prepared for the cloudstack CloudStack deployment.
e. all the hosts in the marvin config are pxe booted via ipmi and cobbler takes over to reimage them with the profile chosen by the jenkins job run.
f. while this is happening we also seed the secondary storage with the systemvm template reqd for the hypervisor.
g. all the primary stores in the marvin config are then cleaned for the next run.

...

9. Puppet slaves within the machines in the environment reach out to puppetmaster to get their identity. mgmt server vm fetches its own recipe and starts configuring itself while hypervisors will do the same in case they need to be acting as kvm KVM agents.

10. When the essential ports for mgmt server - 8080 and 8096 are open and listening we know that the mgmt server has come up successfully. We then go ahead and deploy the configuration specified by marvin.

...

future

• not everyone deploys cloudstack CloudStack the same
• multiple hv environments with multiple hv configurations
• multiple storage configurations

...