Secure KVM VM Live Migration

Bug Reference

Unable to render Jira issues macro, execution error.

Overview

The current VM migration is performed over unencrypted TCP port using the URI scheme qemu+tcp:// and this can allow snooping adversaries to read the VM's state (memory and states) and metadata. With the acceptance of the new CA framework in CloudStack, we can use the framework and sub-system to enable secured live VM migration across KVM hosts.

Feature Specification

After a KVM host is secured by the CA framework, the following files are created in its /etc/cloudstack/agent directory:

cloud.ca.crt: The CA certificate bundle

cloud.crt: The KVM host certificate

cloud.key: The KVM host private key

cloud.csr: The CSR file

cloud.jks: The Java keystore file (the passphrase of which is stored in agent.properties file)

These certificates can be in turn be used to configure TLS for libvirtd while adding a new KVM host using the cloudstack-setup-agent script that already configures libvirtd.conf with following: 

listen_tcp=1
tcp_port="16509"
auth_tcp="none"
listen_tls=0

Such additional configuration will be bound by a (cluster scope) global setting "kvm.libvirtd.tls.enabled", when set to true will perform the following during host addition via the cloudstack-setup-agent script by passing an additiona flag such as "-s" or "--securelibvirtd":

• In the /etc/libvirt/libvirtd.conf (and similar on other non-centos distros) file, the paths to CA, server/client, key files are already assumed to perform minimal changes to the config file we can create some symlinks if the source files exist at /etc/cloudstack/agent path:

ln -s /etc/cloudstack/agent/cloud.ca.crt /etc/pki/CA/cacert.pem
mkdir -p /etc/pki/libvirt/private
ln -s /etc/cloudstack/agent/cloud.crt /etc/pki/libvirt/clientcert.pem
ln -s /etc/cloudstack/agent/cloud.crt /etc/pki/libvirt/servercert.pem
ln -s /etc/cloudstack/agent/cloud.key /etc/pki/libvirt/private/clientkey.pem
ln -s /etc/cloudstack/agent/cloud.key /etc/pki/libvirt/private/serverkey.pem

For reference, the following is seen by default in libvirtd.conf:

#key_file = "/etc/pki/libvirt/private/serverkey.pem"
#cert_file = "/etc/pki/libvirt/servercert.pem"
#ca_file = "/etc/pki/CA/cacert.pem"

• In addition to these changes, remove "listen_tls=0" set by cloudstack-setup-agent. TLS certificate based auth will be enabled therefore explicit authentication using something sasl is not necessary but may be done out-of-band by an admin, by default we'll limit settings to: auth_tcp="none" (this is the current default) and auth_tls="none" (this will be added if global setting was true) in libvirtd.conf.
• Port 16514 needs to be allowed/enabled by firewall/iptables rules, suitable changes will be done in cloudstack-setup-agent and other relevent files.

Proposed Changes

Changes in (KVM) agent:

• Add a new flag such as "-s" in cloudstack-setup-agent that automates the symlinking and additional setup of TLS in libvirtd.conf. The other requirement is that the source cert/pem files already exist at /etc/cloudstack/agent.
• Libvirtd-tls secured hosts should write a config such as libvirtd.tls.enabled=true in agent.properties file, this will be used by the agent to tell management server that it can support qemu+tls:// during migration.
• When certificates are renewed keystore-cert-import may be passed a flag to automate the above (wrt configuring libvirtd and reload/restarting it)

Changes in CloudStack management server and Libvirt computing resource:

• Introduce a new global setting "kvm.libvirtd.tls.enabled" with Scope.Cluster to enable TLS for KVM hosts (for live migration etc), this will be enabled by default. We chose to enable this by default as CA framework generates certs during addition of new hosts using the addHost API, and the cloudstack-setup-agent with "-s" flag will nonetheless check that the certificates actually exists before configuring libvirtd.conf. This setting can also be disabled for a cluster or globally. This will be mentioned in the release notes.
• Changes in keystore-cert-import will be suitably made to additionally reload/restart libvirtd on certificate changes (we need to check if this causes any issues, when agent is already running – this may be done via the agent when its internal task queue is 0)
• When a KVM agent connects to the management server, it will share its host details whether it has its libvirtd TLS setting enabled via the StartupRoutingCommand. And this information will be persisted in host_details.
• When a VM migration is requested, the MigrateCommand will contain the tls enabled setting for source and destination hosts which this will cause the MigrateCommand handler to include the URI to use for migration, i.e. "qemu+tls://" if both source and destination hosts have libvirtd with TLS enabled, otherwise for backward compatibility qemu+tcp:// will be used (a warning may be logged when qemu+tcp:// will be used).
• VM migration can fail (with errors such as CA.pem not found etc) when one of the hosts is not TLS enabled but in agent.properties it says it is (in case of any out-of-band changes) such an error will be logged and may be exposed via the migrateVirtualMachine API.

Upgraded/mixed CloudStack environments:

• After upgraded, existing KVM hosts may not have CA-frameworks and certificates generated and will not have the kvm.libvirtd.tls.enabled define and will assume this to be false.
• Post-upgrade, newly added KVM hosts will have libvirtd configured to use TLS as the global setting will be true (globally or at the cluster level)
• On-boarding strategy can be:
  • Keep the global setting as true to enable tls on kvm/libvirtd for the cluster or globally
  • Execute provisionCertificate API and provide the host id for all hosts and this will create certificates as well as configure libvirtd/tls (if the global setting is enabled) via the keystore-cert-import script.

References

https://wiki.libvirt.org/page/TLSSetup

https://wiki.libvirt.org/page/VNCTLSSetup