Currently all releases of Cloud-platform are based out of ACS master, so it is imperative to monitor and improve the health of ACS master.
For this we need to have infrastructure and automated way of testing ACS master in a continuos fashion. This can be achieved by building a
continous integration system capable which is capable of running tests and creating test beds on demand.
The purpose of the CIS is to provide a means to test and monitor the the health of ASC master. This in turn will help in stabilising Cloud-platform
which is built out of ACS master.
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Marvin+-+Testing+with+Python
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Simulator+integration
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Cloudstack+-+Continuous+Integration
CI : Continuos integration.
CS: CloudStack
CIS: Continuos Integration system.
MTF: Marvin test framework.
MCF: Marvin config file.
Use cases
The CIS is required to create
A fully working test bed, run tests and publish the results.
Provide a means to access test logs for debugging purposes.
Dynamically pull newly added test cases into the CI runs.
Provide a mens to separate tests that can be run using CS simulator.
CIS will create a test setup comprising of single management server supporting one hypervisor at a time. It will be a development
environment. Currently we do not support clustered management server setups.
On successful creation of MS VM it will add it to the database and try to reuse it on subsequent test runs. The reuse of MS VM means
we will not create new VM and install dev tools every time, we will always tear down the old CS setup and create a new one with a new
built form the latest code base.
CIS will work based on a static configuration file added to its database, the configuration file will be in accordance with the CS
(marvin test framework) MTF. It may contain additional details that are not required by MTF but are used by CIS system itself.
As of now we do not pick the resources dynamically and generate the MCF. It has to be created manually and
added to the CIS database. There is no validation of the MCF agains other MCFs that may be present in the CIS database. we
have to make sure that we do not add overlapping resources in MCF files.
CIS will use only open source tools to function. Currently we will use Jenkins, Cobbler and Puppet to create CS test beds.
It will run tests and publish the results in mail and archive the test results and logs on a nfs server.
The CIS database is a mysql database located on the driver VM. We need to have access to the driver VM and the database to configure CIS.
Create a MCF and add its path to the static_config table of the CIS database.
Add the details (IPMI address , MAC etc) of hosts mentioned in the above MCF file to the static_host_config table of CIS database.
Add the location and version of system vm templates required to build the test bed described in the above mentioned MCF. The version of
system vm templates is just the name of the brach from which we are building the CS MS.
After each run the CIS system make the detailed test report available via jenkins reporter job. These jobs will stay in the Jenkins server indefinitely
unless removed. The CIS can be configured to cleanup these jobs at specified interval. We need to set the self.deleteAfterDays value in the file jenkinsGarbageCollector.py.
we need to add this as cron job and run it regularly to cleanup the report generator jobs in jenkins.
Log into Jenkins, create a Automation CI view and add all the CIS related jobs to this view. The CIS related jobs can be crated from the xml file located
in the CIS source code.
Configure the Automation CI trigger job to schedule the CI runs. This job will trigger the CI run jobs on each of the hypervisor.
There is a job associated to run the CI on each type of hypervisor. Add the default values of build parameters and save the config.
The default parameters will be used to for the daily automated CI runs.
Configure the test executor job by specifiying the number of test that can be run in parallel.
Once all the jobs are in place, Enable these jobs. CIS system should now be completely operational.
Jenkins: This is an open source application that schedules and monitors jobs.
Driver VM: This is the core component of the CI system. This VM handles creation of management server, refreshing the hosts, deploying data center, keeping track of the
resources and execution of tests.
Infra xen server cluster: These are set of xen servers which are a part of the CI system and are used for hosting management servers and running other required service like a nfs servers , secondary DNS servers etc.
Server farm: These are the set of machines used as hypervisors which are used to create cloudstack datacenter.
There are other components of the CIS like a proxy server, a open filer VM which are not shown in the above diagram.
Jenkins, puppet, cobbler, marvin, python virtual environment.