The information here will be of interest to people looking to understand, improve and replicate Apache CloudStack test infrastructure across sites.
This document talks about the *evolving* continuous test infrastructure used to
setup, deploy, configure and test Apache CloudStack. Information here is useful
for anyone involved in build, test, continuous integration and operators of
CloudStack.
components
. nightly yum/apt repositories
. cobbler
.. rhel / ubuntu/ debian kickstarts
.. hypervisor kickstarts
.. adding new profiles
. puppet
. dnsmasq
. jenkins jnlp slave
. scheduling
This document talks about the continuous test infrastructure used to
setup, deploy, configure and test Apache CloudStack. Information here is useful
for anyone involved in build, test, continuous integration and operators of
CloudStack.
. nightly yum/apt repositories
. cobbler
.. rhel / ubuntu/ debian kickstarts
.. hypervisor kickstarts
.. adding new profiles
. puppet
. dnsmasq
. jenkins jnlp slave
. scheduling
\[insert diagram here\] |
The above illustration shows a high-level view of the test infrastructure
setup. The workflow listed below shows how this setup works and the tools
interacting to make that possible
1. At the center of the workflow is the "driver" appliance that manages the
infrastructure. This is a Cent OS 6.2 VM running on a XenServer. The
"driver"-VM is responsible for triggering the process when it is time for a
test run.
The driver appliance is composed of the following parts:
a. Cobbler - cobbler is a provisioning PXE server (and much more) useful for
rapid setup of linux machines. It can do DNS, DHCP, power management and package
configuration via puppet. It is capable of managing network installation of
both physical and virtual infrastructure. Cobbler comes with an expressive CLI
as well as web-ui frontends for management.
Cobbler manages installations through profiles and systems:
profiles - these are text files called kickstarts defined for a distribution's
installation. For eg: RHEL 6.1 or Ubuntu 12.04 LTS. Each of the machines in the
test environment - hypervisors and cloudstack management servers contains a
profile in the form of a kickstart.
The profile list looks as follows:
\[root@infra \~\]# cobbler profile list |
cloudstack-rhel
cloudstack-ubuntu
rhel63-kvm
rhel63-x86_64
ubuntu1204-x86_64
xen602
xen56
systems - these are virtual/physical infrastructure mapped to cobbler profiles
based on the hostnames of machines that can come alive within the environment.
\[root@infra \~\]# cobbler system list |
acs-qa-h11
acs-qa-h20
acs-qa-h21
acs-qa-h23
cloudstack-rhel
cloudstack-ubuntu
When a new image needs to be added we create a 'distro' in cobbler and
associate that with a profile's kickstart. Any new systems to be hooked-up to
be serviced by the profile can then be added easily by cmd line.
b. Puppet master - Cobbler reimages machines on-demand but it is upto puppet
recipes to do configuration management within them. The configuration
management is required for kvm hypervisors (kvm agent for eg:) and for the
cloudstack management server which needs mysql, cloudstack, etc. The
puppetmasterd daemon on the driver-vm is responsible for 'kicking' nodes to
initiate configuration management on themselves when they come alive.
So the driver-vm is also the repository of all the puppet recipes for various
modules that need to be configured for the test infrastructure to work. The
modules are placed in /etc/puppet and bear the same structure as our github
repo. When we need to affect a configuration change on any of our systems we
only change the github repo and the systems in place are affected upon next
run.
c. dnsmasq - DNS is controlled by cobbler but its configuration of hosts is set
within dnsmasq.d/hosts. This is a simple 1-1 mapping of hostnames with IPs. For
the most part this should be the single place where one needs to alter for
replicating the test setup. Everywhere else only DNS names are/should-be used.
d. dhcp - DHCP is also done by dnsmasq. All configuration is in
/etc/dnsmasq.conf. static mac-ip-name mappings are given for hypervisors while
the virtual instances get dynamic ips
e. ipmitool - ipmi for power management is setup on all the test servers and
the ipmitool provides a convienient cli for booting the machines on the network
into PXEing.
f. jenkins-slave - jenkins slave.jar is placed on the driver-vm as a service in
/etc/init.d to react to jenkins schedules and to post reports to. The slave
runs in headless mode as the driver-vm does not run X.