Bug Reference

Unable to render Jira issues macro, execution error.

Introduction

With the introduction of Secure Agent Communications in 4.11, it is only limited to securing of how host's agent connects to the management server. By default, the libvirtd process listens on tcp and therefore any VM migration from/to such a host is insecure. The same X509 certificates used by the agent provisioned by the CA framework can be used to secure libvirtd process both during the addition of a KVM host and during provisioning/renewal of certificates.

Feature Specifications

After a KVM hosts communication with the management server is secured by the CA framework, by either using the addHost API or out-of-band using the keystore scripts and issueCertificate API, the following files are created in it's /etc/cloudstack/agent directory:
• cloud.ca.crt: The CA certificate bundle
• cloud.crt: The KVM host certificate
• cloud.key: The KVM host private key
• cloud.csr: The CSR file
• cloud.jks: The Java keystore file (the passphrase of which is stored in agent.properties file)

These certificates can, in turn, be used to configure TLS for libvirtd, while adding a new KVM host using the cloudstack-setup-agent script that already configures libvirtd.conf with the following settings:
listen_tcp=1
tcp_port="16509"
auth_tcp="none"
listen_tls=0

Once Libvirtd is setup with TLS, secure live VM migration are possible by using the URI qemu+tls:// with the requirement that both source and destination hosts should have TLS set up.

Architecture and Design description

During a host addition (addHost) and provisionCertificate API actions, the KVM host agent will determine if the necessary certificates are in place such that secure (tls) migrations are possible. If they are, then the host agent will configure TLS for libvirtd using the cloudstack-setup-agent.

Libvirtd configuration

The libvirtd config file such as at /etc/libvirt/libvirtd.conf, will receive TLS configuration using the assumed path and symlinks will be created as the following:

ln -s /etc/cloudstack/agent/cloud.ca.crt /etc/pki/CA/cacert.pem

mkdir -p /etc/pki/libvirt/private

ln -s /etc/cloudstack/agent/cloud.crt /etc/pki/libvirt/clientcert.pem

ln -s /etc/cloudstack/agent/cloud.crt /etc/pki/libvirt/servercert.pem

ln -s /etc/cloudstack/agent/cloud.key /etc/pki/libvirt/private/clientkey.pem

ln -s /etc/cloudstack/agent/cloud.key /etc/pki/libvirt/private/serverkey.pem

For reference, the following is seen by default in libvirtd.conf:

#key_file = "/etc/pki/libvirt/private/serverkey.pem"

#cert_file = "/etc/pki/libvirt/servercert.pem"

#ca_file = "/etc/pki/CA/cacert.pem"

In addition to these changes:

• The "listen_tls" will be set to "listen_tls=1" by the cloudstack-setup-agent script when TLS is to be enabled.

• TLS certificate based auth will be enabled, therefore explicit authentication using something like Simple Authentication and Security Layer (SASL) is not necessary and may be done out-of-band by an admin. By default we'll limit settings to: 

• auth_tcp="none" (the current default)

• auth_tls="none"

• Port 16514 needs to be allowed/enabled by firewall/iptables rules, suitable changes will be done in cloudstack-setup-agent and other relevent files.

Agent changes

• A secured host will imply that both agent and libvirtd has certificates setup. When the keystore or certificates files are not available, host will be assumed as unsecured. This will be used by the agent and libvirt resource on whether qemu+tls:// can be used during live VM migration.

• When certificates are renewed, keystore-cert-import will be used to reconfirm libvirt’s security configuration.

• Subject to testing of service/libvirtd unavailability, a restart task may be performed when the agent has no other outstanding task.

Management Server changes

• During addition of a host via the addHost API, or after an provisionCertificate API call, the cloudstack-setup-agent will be passed a configuration flag to ask it to setup and enable tls for libvirtd. 

• When a KVM agent connects to the management server, it will share its host details whether it has its libvirtd TLS setting enabled via the StartupRoutingCommand. And this information will be persisted in host_details.

• When a VM migration is requested, the MigrateCommand handler will determine which URI to use based on availability of certificates.

• When VM migration fails due to certificate related issues (for example CA.pem not found or TLS handshake failure), such errors will be logged and may be exposed to the admin throught the migrateVirtualMachine API/error response.

• MigrateVirtualMachine API failure due to migration between non-secured and secured host will be logged and may be exposed as error in the api response.

API changes

N/A

DB changes

N/A

UI changes

Show unsecured KVM hosts in the host list.

Provide a button in host's detail/view to provision certificate to the KVM host.

During a host addition (addHost) and provisionCertificate API actions, the KVM host agent will determine if the necessary certificates are in place such that secure (tls) migrations are possible. If they are, then the host agent will configure TLS for libvirtd using the cloudstack-setup-agent.
3.5.1  Libvirtd configuration
The libvirtd config file such as at /etc/libvirt/libvirtd.conf, will receive TLS configuration using the assumed path and symlinks will be created as the following:
ln -s /etc/cloudstack/agent/cloud.ca.crt /etc/pki/CA/cacert.pem
mkdir -p /etc/pki/libvirt/private
ln -s /etc/cloudstack/agent/cloud.crt /etc/pki/libvirt/clientcert.pem
ln -s /etc/cloudstack/agent/cloud.crt /etc/pki/libvirt/servercert.pem
ln -s /etc/cloudstack/agent/cloud.key /etc/pki/libvirt/private/clientkey.pem
ln -s /etc/cloudstack/agent/cloud.key /etc/pki/libvirt/private/serverkey.pem
For reference, the following is seen by default in libvirtd.conf:
#key_file = "/etc/pki/libvirt/private/serverkey.pem"
#cert_file = "/etc/pki/libvirt/servercert.pem"
#ca_file = "/etc/pki/CA/cacert.pem"
In addition to these changes:
•  The "listen_tls" will be set to "listen_tls=1" by the cloudstack-setup-agent script when TLS is to be enabled.
•  TLS certificate based auth will be enabled, therefore explicit authentication using something like Simple Authentication and Security Layer (SASL) is not necessary and may be done out-of-band by an admin. By default we'll limit settings to:
•  auth_tcp="none" (the current default)
•  auth_tls="none"
•  Port 16514 needs to be allowed/enabled by firewall/iptables rules, suitable changes will be done in cloudstack-setup-agent and other relevent files.
3.5.2  Agent changes
•  A secured host will imply that both agent and libvirtd has certificates setup. When the keystore or certificates files are not available, host will be assumed as unsecured. This will be used by the agent and libvirt resource on whether qemu+tls:// can be used during live VM migration.
•  When certificates are renewed, keystore-cert-import will be used to reconfirm libvirt’s security configuration.
•  Subject to testing of service/libvirtd unavailability, a restart task may be performed when the agent has no other outstanding task.
3.5.3  Management Server changes
•  During addition of a host via the addHost API, or after an provisionCertificate API call, the cloudstack-setup-agent will be passed a configuration flag to ask it to setup and enable tls for libvirtd.
•  When a KVM agent connects to the management server, it will share its host details whether it has its libvirtd TLS setting enabled via the StartupRoutingCommand. And this information will be persisted in host_details.
•  When a VM migration is requested, the MigrateCommand handler will determine which URI to use based on availability of certificates.
•  When VM migration fails due to certificate related issues (for example CA.pem not found or TLS handshake failure), such errors will be logged and may be exposed to the admin throught the migrateVirtualMachine API/error response.
•  MigrateVirtualMachine API failure due to migration between non-secured and secured host will be logged and may be exposed as error in the api response.