Implementation Details of uploading custom certificate

API - UploadCustomCertificate 

• API has to be invoked multiple times to upload the certificates and has to be done in order following from root cert, 0 or more intermediate certs followed by server cert and private key.
• Uploading root certificate and the server certificates are mandatory steps.
• Uploading intermediate certificates(s) is optional.
• Self-signed certificates are not supported.
• Reverting back to Realhostip.com or the last certificate is not supported.

DataBase - table storing the certificates 'keystore'

Schema - 

mysql> desc keystore;
+---------------+---------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+---------------+---------------------+------+-----+---------+----------------+
| id | bigint(20) unsigned | NO | PRI | NULL | auto_increment |
| name | varchar(64) | NO | UNI | NULL | |
| certificate | text | NO | | NULL | |
| key | text | YES | | NULL | |
| domain_suffix | varchar(256) | NO | | NULL | |
| seq | int(11) | YES | | NULL | |
+---------------+---------------------+------+-----+---------+----------------+

SSVM

• SecStorageSetupCommand -
  • All ssvms are rebooted when UploadCustomCertificate is invoked for server cert and private key. As part of ssvm agent connect with MS, a command called SecStorageSetupCommand containing all the certs and the key is sent to ssvm (you wont see the certs as it is sensitive information and not logged). 
  • On the ssvm side the root cert is inserted into the keystore. 
  • For apache server a server cert. file, a key file and a certificate chain file (consisting of certs from root cert to the intermediate cert) are created in the appropriate location and location put into the config file.   
    
• SSVM acts as a server for download template/volume/iso operations and during copyTemplate if it is in the source zone. It uses an Apache webserver when acting as server.
  • Apache server should have the latest key and server cert under the following location - /etc/ssl/private/cert_apache.key and /etc/ssl/certs/cert_apache.crt

  • /etc/apache2/sites-available/default-ssl is the config file for apache server where the chain location, server cert and private key are configured.

  • When client contacts the apache server it needs to present the entire chain of certificates to the client.
• It acts as a client during copyTemplate operation if it is in the destination zone. SSVM uses a Java client when acting as a client.
  • SSVM has a java keystore which should have the root certificate in the keystore so that when it tries to download from the url it can validate the chain presented by the server against the known root.
    Use the command below to list all certs in the keystore
     keytool -list -keystore /usr/local/cloud/systemvm/certs/realhostip.keystore -storepass vmops.com

CPVM

• CPVM acts as a server only and uses Java HTTP server

TroubleShooting

1. Download urls point to the old domain.
   1. Reduce the expiration duration of the urls by changing global config extract.url.expiration.interval 
   2. And change the frequency for cleanup thread through extract.url.cleanup.interval restart MS. 
   3. Wait for the cleanup thread duration and try downloading again. See whether the url is deleted.
2. CopyTemplate giving the exception - PKIX certification error
   1. Via browser or any client, check whether Apache Webserver for SSVM is sending the entire chain
   2. Check whether the realhostip.keystore (Java keystore) on SSVM has the root CA certificate : keytool -list -keystore realhostip.keystore -storepass vmops.com
3. Download urls are not working. 
   1. Via browser or any client, check whether Apache Webserver for SSVM is sending the entire chain

4. I don’t see the latest systemvm.iso being patched since the /usr/local/cloud/systemvm/config_ssl.sh  on ssvm is still the old one

5. I changed CPVM to work on HTTPS from HTTP, or vice-versa. It does not change.
   1. You may need to destroy and recreate your CPVM when switching between HTTP and HTTPS protocols
6. Certificate encoding - 
   1. Make sure you have correctly url encoded the certificates. To do this you can check cloud database and check the entries in keystore table and match them with the original certificates you had.
7. Uploaded wrong certificate -
   1. In case you uploaded wrong certificates for root/intermediate, you can undo that by calling the api with same name and domainsuffix. In case you uploaded wrong server certificate upload the right one through the UI keeping the same domain suffix.
8. Any other issues with ssvm -  
   
   1. Check logs - Check MS logs (/var/log/cloudstack/management ) and ssvm logs (under /var/log/cloud/) to see if you can find any exceptions in programming the certificate.
   2. Destroy ssvm - If nothing is apparent from the logs try destroying ssvm as one of the last resorts. Try this in one of the zones and see if the new ssvm which comes up solves the issues.
9. I have already uploaded certificate and chain in a prior CCP version. Am I all good?
   1. Unfortunately, no. You will need to re-upload the whole chain using the same API parameters (name, id etc.)