I have tried and put some troubleshooting tips which hopefully should help you get out of troubled waters. But, this has to be a collective community effort so in case you find some more info or something which is not already covered please add to it so that we can learn from each other.
- Logging into ssvm through the hypervisor "ssh -i /opt/xensource/bin/id_rsa --p 3922 root@privateIP_or_LinkLocalIpofSSVM", or "ssh -i /root/.ssh/id_rsa.cloud -p 3922 root@LinkLocal" on XenServer. Private ip in case of vmware and linklocal in case Xenserver.
- SSVM health check - Run the following script inside ssvm /usr/local/cloud/systemvm/ssvm-check.sh
It checks for 1)connectivity with DNS server 2) resolving of domain names 3)status of secondary storage 4)ability to write to secondary storage 5)connectivity with management server at port 8250 and 6) status of java process. - Template not ready / not available when creating an instance - Many a times the SSVM is running but still the templates do not show as ready or to say templates are not available when creating an instance. Run the health check script above and diagnose. The most probable reason reason is that the agent running on SSVM hasn't been able to connect with MS which could also be validated by checking the host table in DB. select * from host where type like 'SecondaryStorageVM'. If the status shows as Alert then definitely that is the reason. There could be a number of reasons for the agent not being able to connect with MS. Below three could be one of them.
- Check whether port 8250 is open on MS and there is no firewall rule. This is the port on which the agent and MS communication happens.
- Check whether the SSVM is trying to connect to the right ip of MS. If it is incorrect it could be due to the wrong ip being set in the global settings (configuration table) for 'host' in MS. Change that, restart MS and SSVM and see if it solves the issue.
- Check the agent status on SSVM- See if the agent is running by typing "service cloud status" in SSVM. Try to run it and see if that's successful or changes the alert status.
- To check the state of templates whether is has downloaded or there is an error - Log into DB and check table template_host_ref and observe the download_state and error_string.
- Templates stuck in download in progress - Either stop and then start the SSVM. Or, run service cloud restart on the SSVM. You can also restart MS. This would trigger template sync which essentially will try and resume such stuck templates or redo the download of erred out templates
- Connection refused as the status for the template - Check whether the config parameter "secstorage.allowed.internal.sites" has been set to allow the internal n/w URL's.
- Retrying the download of templates - Try restarting MS / SSVM.
- SSVM Logs - /var/log/cloud/cloud.log
- SSVM templates physical location - find the mount point by typing command "mount" . Go to the directory and under template/tmpl you will find all the templates.
- SSVM Apache server - For 2.2 onwards the system vms are debian based. Type "service apache2 status" to find the status. Apache root is at /www/html/
- Run script of java process /usr/local/cloud/systemvm/run.sh
- Increasing log level - 1) Edit the file /usr/local/cloud/systemvm/conf/log4j-cloud.xml 2) For the log file cloud.log change the threshold to info: <param name="Threshold" value="WARN"/> to <param name="Threshold" value="INFO"/> 3) Change com.cloud to INFO: <category name="com.cloud"> <priority value="INFO"/> </category> If you're not getting sufficient logging, you can also try setting it to DEBUG.
- Download Complete 100% but getting error like this Failed post download script: /usr/sbin/vhd-utilvhd tool check /mnt/SecStorage/33e2e9f5/template/tmpl/345/447/dnld1469110483936142751tmp_ failed - Many reasons for this but amongst them are wrong OS selection, vhd corruption.
Test this in the lab by copying the template to one of the hosts then on that host run
vhd-util check -n filename.vhd
vhd-util scan filename.vhd - SSVM RAM - Set the param secstorage.vm.ram.size to in change the ram size of the vm. Default in the code is 256.
- Allow multiple secondary storages feature has been added in the 2.2.x series. This would help in scaling the secondary storages for snapshots. The private templates are copied to one of the secondary storages and public to all of them. The template sync happens only for public templates.
- For each secondary storage there is a corresponding row created in the host table.
- HTTP Server returned 403 (expected 200 OK) - For copy templates.
Try to see the first log for this template initiation ? It should be logged with DownloadCommand and should have the url of the source ssvm's template. Then you can try going to the destination SSVM and try downloading that url.
See what issues you get. I would also check the iptable rules to see if the destination ssvm is blocked from accessing the source ssvm and also if there is any .htaccess file in the apache directories forbidding the download of template
One of the problems as was as follows.
The problem is that we're using basic networking & have the private network setup with the same gateway & subnet as the public network. When the storage VM comes up the public network gets setup first but then when the private network comes up on eth2 it clobbers the gateway & sets it to the eth2 interface. So when the copy is initiated between the storage VMs it happens across the private network but the /var/www/html/copy/.htaccess file only allows the public IP of the other SSVM, thus the 403 errors.
/