• Created by Unknown User (tsp), last updated on Jan 24, 2013 3 minute read

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 7 Next »

The information here will be of interest to people looking to understand, improve and replicate Apache CloudStack test infrastructure across sites. 

about

This document talks about the *evolving* continuous test infrastructure used to setup, deploy, configure and test Apache CloudStack. Information here is useful for anyone involved in build, test, continuous integration and operators of CloudStack.

components

. nightly yum/apt repositories

. cobbler

    .. rhel / ubuntu/ debian kickstarts

    .. hypervisor kickstarts

    .. adding new profiles

. puppet

. dnsmasq 

. jenkins jnlp slave

. scheduling

[insert diagram here]

The above illustration shows a high-level view of the test infrastructure setup. The workflow listed below shows how this setup works and the tools interacting to make that possible

1. At the center of the workflow is the "driver" appliance that manages the infrastructure. This is a Cent OS 6.2 VM running on a XenServer. The "driver"-VM is responsible for triggering the process when it is time for a test run.

The driver appliance is composed of the following parts:

a. Cobbler - cobbler is a provisioning PXE server (and much more) useful for rapid setup of linux machines. It can do DNS, DHCP, power management and package configuration via puppet. It is capable of managing network installation of both physical and virtual infrastructure. Cobbler comes with an expressive CLI as well as web-ui frontends for management.

Cobbler manages installations through profiles and systems:

profiles - these are text files called kickstarts defined for a distribution's installation. For eg: RHEL 6.1 or Ubuntu 12.04 LTS. Each of the machines in the test environment - hypervisors and cloudstack management servers contains a profile in the form of a kickstart.  

The profile list looks as follows:

[root@infra ~]# cobbler profile list

   cloudstack-rhel

   cloudstack-ubuntu

   rhel63-kvm

   rhel63-x86_64

   ubuntu1204-x86_64

   xen602

   xen56

systems - these are virtual/physical infrastructure mapped to cobbler profiles based on the hostnames of machines that can come alive within the environment.

[root@infra ~]# cobbler system list

   acs-qa-h11

   acs-qa-h20

   acs-qa-h21

   acs-qa-h23

   cloudstack-rhel

   cloudstack-ubuntu

When a new image needs to be added we create a 'distro' in cobbler and associate that with a profile's kickstart. Any new systems to be hooked-up to be serviced by the profile can then be added easily by cmd line.

b. Puppet master - Cobbler reimages machines on-demand but it is upto puppet recipes to do configuration management within them. The configuration management is required for kvm hypervisors (kvm agent for eg:) and for the cloudstack management server which needs mysql, cloudstack, etc. The puppetmasterd daemon on the driver-vm is responsible for 'kicking' nodes to initiate configuration management on themselves when they come alive. 

So the driver-vm is also the repository of all the puppet recipes for various modules that need to be configured for the test infrastructure to work. The modules are placed in /etc/puppet and bear the same structure as our github repo. When we need to affect a configuration change on any of our systems we only change the github repo and the systems in place are affected upon next run.

c. dnsmasq - DNS is controlled by cobbler but its configuration of hosts is set within dnsmasq.d/hosts. This is a simple 1-1 mapping of hostnames with IPs. For the most part this should be the single place where one needs to alter for replicating the test setup. Everywhere else only DNS names are/should-be used.

d. dhcp - DHCP is also done by dnsmasq. All configuration is in /etc/dnsmasq.conf. static mac-ip-name mappings are given for hypervisors while the virtual instances get dynamic ips

e. ipmitool - ipmi for power management is setup on all the test servers and the ipmitool provides a convienient cli for booting the machines on the network into PXEing.

f. jenkins-slave - jenkins slave.jar is placed on the driver-vm as a service in /etc/init.d to react to jenkins schedules and to post reports to. The slave runs in headless mode as the driver-vm does not run X.

workflow

marvin integration

control via github

limitations

- not everyone deploys cloudstack the same

- multiple hv environments with multiple hv configurations

- multiple storage configurations

enhancements

- dogfooding

- logging enhancements = archiving + rsyslog

- digest emails via jenkins. controlling spam

- external devices (LB, VPX, FW) 

- mcollective?

troubleshooting

acknowledgements

  • No labels