You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 26 Next »

Introduction

Linux Containers (LXC) is a lightweight system virtualization that uses resource isolation instead of the hardware emulation approach used by KVM and Xen. For users who do not require full OS virtualization as provided by KVM and Xen, container technologies such as LXC provide an attractive performant solution for virtualization.

Purpose

This docoument contains the design specification for LXC support in Cloudstack.

Status

Currently in development/testing, see last section below for details.

References

Feature Specification

LXC will be implemented as a hypervisor in Cloudstack and will be a first class citizen to the other hypervisors such as Xen, KVM, VMWare. As such, a user will be able to select LXC as the hypervisor for all areas a hypervisor is selectable and where the system resources have been met.

Primary storage

Available storage options for LXC primary storage are NFS and SharedMountPoint.

Secondary storage

Unlike other hypervisors where a VM is contained in a single image file, LXC containers run from a directory that serves as the root filesystem. LXC template images will be stored in TAR format in secondary storage. See LXC Templates section for details on how the image is unpacked.

Guest VM creation

Similar to KVM, LXC virtual machines will be created using libvirt. The libvirt domain xml will include two additional elements needed for LXC: <init> and <filesystem>.

<domain type='lxc'>
  <os>
    <type arch='x86_64'>exe</type>
    <!-- specifies the startup script -->
    <init>/sbin/init</init>
  </os>
  <devices>
    <!-- specifies the directory containing the root filesystem -->
    <filesystem type='mount'>
      <source dir='/mnt/primary/edb596f6-42fb-499d-8ded-8834aff52d75'/>
      <target dir='/'/>
    </filesystem>
  </devices>
</domain>

LXC Templates

Downloadable LXC template images should be stored as either tar.gz or tar formats. The SecondaryStorage VM will download and store the template as a tar file.

bash$ find /export/secondary -type f
/export/secondary/template/tmpl/1/10/template.properties
/export/secondary/template/tmpl/1/10/402b0be5-b840-3fef-b292-d330f3bf809a.tar

During the creation of the first VM for an LXC template, the management server will send a PrimaryStorageDownload command to the agent on the LXC host. This command makes a copy of the template from secondary storage onto primary storage. This copy is used as a base for creating all LXC images for the cluster and is not used directly to run a VM. The copy operation from secondary storage to primary storage will unpack the tar file into the destination template directory.

bash$ ls -ld /mnt/primary/*
dr-xr-xr-x. 23 root root      4096 Jan 25 11:33 /mnt/primary/2cc4e71e-2e4b-4987-a48e-dfae08e0d767

bash$ ls /mnt/primary/2cc4e71e-2e4b-4987-a48e-dfae08e0d767
bin   cgroup  etc   lib    media  opt   root  selinux  sys  usr
boot  dev     home  lib64  mnt    proc  sbin  srv      tmp  var

After a copy of the template is available on primary storage, the management server will send a CreateCommand to the LXC host to create a disk from the template. This involves a recursive copy of the template directory to the root directory for the VM.

bash$ ls -ld /mnt/primary/*
     4 dr-xr-xr-x. 23 root root      4096 Jan 25 11:33 2cc4e71e-2e4b-4987-a48e-dfae08e0d767
     4 drwxr--r--. 23 root root      4096 Jan 25 13:27 edb596f6-42fb-499d-8ded-8834aff52d75

System VMs

Each of the different hypervisors currently have their own System VMs. These system VM images are used to run a console proxy, secondary storage, and router VMs.

We discussed the possibility of creating System VMs for LXC. There was concern with the complexity and potential issues involving iptables for the router inside an LXC container. As an intermediate solution we are going to use KVM System VMs inside the LXC Cluster.

Development/testing notes

For minimum requirements, you will need to run the management server and an LXC host. The LXC host has the exact same setup as KVM, the only difference is you specify the "lxc" hypervisor in the agent config file. The LXC host will run the system VMs under KVM.

I am currently running my dev/test setup on two machines: 1) runs the management and 2) the other runs the LXC agent. The following sections detail how to get the development code running so that you can spin up LXC containers with Cloudstack. It's not very elegant, but is the easiest way I've found to quickly get the system up with development code. There's probably an easier way to do this, like with RPMs, but I haven't gotten around to that yet.

1. Install Cloudstack 4.0

  • Install the agent
    • This will install the KVM agent, we will configure it later for LXC

2. Checkout and build latest code

  • Checkout and build the code on both the management server and lxc host machines.
    • git clone git@github.com:gilt/incubator-cloudstack.git
    • cd incubator-cloudstack
    • mvn clean install

3. Update management server with 4.1 code

  • Load the data model
    • cd incubator-cloustack
    • mvn -P developer -pl developer -Ddeploydb
  • Run the management server
    • mvn -pl :cloud-client-ui jetty:run
  • Load LXC data model
    • load cloudstack/setup/db/lxc.sql
  • Import System VM Template
    • Modify /usr/lib64/cloud/common/scripts/storage/secondary/cloud-install-sys-tmplt and add a section for "lxc"
    • if [ "$templateId" == "" ]
then
   if [ "$hyper" == "kvm" ]
   then
      ext="qcow2"
      templateId=(`mysql -h $dbHost --user=$dbUser --password=$dbPassword --skip-column-names -U cloud -e "select max(id) from cloud.vm_template where type = \"SYSTEM\" and\
 hypervisor_type = \"KVM\" and removed is null"`)
<insert below>
   elif [ "$hyper" == "lxc" ]
   then
      ext="qcow2"
      templateId=(`mysql -h $dbHost --user=$dbUser --password=$dbPassword --skip-column-names -U cloud -e "select max(id) from cloud.vm_template where type = \"SYSTEM\" and\
 hypervisor_type = \"LXC\" and removed is null"`)
<end of insert>
   elif [ "$hyper" == "xenserver" ]
    • Run the import script
    • /usr/lib64/cloud/common/scripts/storage/secondary/cloud-install-sys-tmplt -m /mnt/secondary -u http://download.cloud.com/templates/acton/acton-systemvm-02062012.qcow2.bz2 -h lxc -F
  • Restart management server

4. Update LXC agent with 4.1 code

  • Modify /etc/cloud/agent/agent.properties, add:
    • hypervisor.type=lxc
  • Modify /etc/init.d/cloud-agent
    • Replace the "export CLASSPATH" line with
    • PCP=""
for jarfile in <incubator-cloudstack-dir>/client/target/cloud-client-ui-4.2.0-SNAPSHOT/WEB-INF/lib/cloud-*.jar /usr/share/java/jna.jar; do
    if [ ! -e "$jarfile" ] ; then continue ; fi
    PCP=$jarfile:$PCP
done
export CLASSPATH="$SCP:$DCP:$PCP:$JCP:/etc/cloud/agent:/usr/lib64/cloud/agent:<incubator-cloudstack-dir>/client/target/cloud-client-ui-4.2.0-SNAPSHOT/WEB-INF/classes/"
  • Restart cloud-agent
    • service cloud-agent restart

5. Update Secondary Storage System VM with 4.1 code

In this step we bring up the system to the point where system VMs are created. After the Secondary Storage VM starts up, we will swap out the cloud jars with the ones we've built.

  • From the Cloudstack UI: add a zone, pod, cluster, host, primary, and secondary for LXC
  • After the Secondary Storage VM starts up, from the lxc host: 
    • ssh -i /root/.ssh/id_rsa.cloud -p 3922 root@169.254.x.x (SSVM local link ip)
cp -r /usr/local/cloud/systemvm /usr/local/cloud/systemvm.orig
rm /usr/local/cloud/systemvm/cloud-*.jar
scp root@<lxc-host>:<incubator-cloudstack-dir>/client/target/cloud-client-ui-4.2.0-SNAPSHOT/WEB-INF/lib/cloud-*.jar /usr/local/cloud/systemvm
service cloud stop
service cloud start

6. Creating LXC instances

  • Wait for the SSVM to download the Centos 6.3 LXC image
  • Create an LXC instance
  • Login with cloud/cloud
  • No labels