Page History

...

• Explore debootstrap, nbd and chroot based systemvm template building. PoC shows promise, builds template under 5mins. (vs 30-60mins before), refer to PoC tree for export logic too.
• Instead of bundling all 3x systemvmtemplates, we can bundle one of the hypervisors and build/export the image for other hypervisors using qemu-img upon cloudstack-management pkg install/upgrade (we use qemu-img to convert the template to different hypervisor images during build too):
  • Bundle qcow2 file in cloudstack-management pkg
  • Use export mechanism that uses qemu-img, tar and bzip2 to export to other hypervisor formats - ova and vhd. Check for qemu-img compatibility/feasibility for all support mgmt server distros. This can be done during automated template upgrades/updates or when cloudstack-management pkg is installed or upgraded.
  • Assume we bundle OVA/vmware; upon install/upgrade we extract the ova (tar) and grab the vmdk to export for KVM and XenServer.
  • To export for kvm, we run: qemu-img -f vmdk -O qcow2 -o compat=0.10 -c <source vmdk> <dest kvm qcow2 file>
  • To export for xenserver/xcp-ng, we run: qemu-img -f vmdk -O vpc <source vmdk> <dest xs/xcp vhd file>
  • In testing, we found this entire process of conversion takes 10-30seconds in total; which may significantly save on bandwidth of over 1G in total cloudstack-management pkg/file being downloaded
  • The logic/script that does this would check and do this kind of convertion only if qcow2 and vhds are not bundled with the cloudstack-management pkg
• During packaging, this can simplify things so if systemvm.iso is deprecated then we bundle the ova only when -D systemvm or -P systemvm is enabled

...

• Remove systemvm.iso
  • With ssh/scp setup via public ssh key of mgmt server, sent to cmdline string; we don't need systemvm.iso.
  • Deprecate and remove systemvm.iso file, move systemvmtemplate patching logic via ssh/scp.
  • Don't remove ISO drive in systemvm VM spec (so can be used in future by config drive etc)
• SSVM, CPVM CA/cert patching changes:
  • Using CA framework create certificate and pass them (pub/priv, ca cert) as key/value params in the cmdline string for all systemvms and VRs. This is do-able if we know the hostname and IP address (esp the private and link-local nic) of the systemvms/VRs.
  • The certificates can be used in future by VR agent, and currently by SSVM/CPVM agents
  • Deprecate and remove CSR-based ssvm/cpvm certificate setup method
• Live and Automatic Patching:
  • Once ssh/scp based patching is implemented and there exists internal Java Service/APIs to trigger this; why can't we do this on the fly too for example during minor-release upgrades or upgrades where systemvmtemplate is not upgraded.
  • A new script/tool/utility to facilitate live patching (the tool is scp'd to systemvm/VR too).
  • It may also be useful to introduce a new live API or extend current API such as upgradeRouterTemplate with a new live=true|false parameter. This can allow root-admins to manually select and live-upgrade their routers without restarting network with cleanup=true.
  • Explore automatic upgrade/patching of routers, which can be triggered via some checksum-checks; or simply when restartNetwork is triggered without cleanup (i.e. VR is not destroyed) but with a new parameter to enable/allow live patching of scripts/software.
  • Live patching mechanism to explore pre/post patch and validation hooks, ability to restore upon patching failure (heal & failsafe)
  • If live patching uses ssh, can it be used to patch when upgrade older systemvms/systemvmtemplates and VR:
    • ssvm/cpvm: their internal nics, config etc hasn't changed in ages, in general, all it requires is (a) update the jars, (b) update certs incl ca-certificates, (c) updating maybe the JRE and (d) restart the cloud process
    • routers: their internal nics order hasn't changed in ages, in general it may require (a) update cloud scripts, (b) restarting all enabled services; however between major ACS versions the userspace software (haproxy, apache, dnsmasq etc) has significantly changes so those may not be guaranteed to work.
• New Multi-hop ssh based communication Utility:A standard Investigate and explore multi-hop ssh utility class/client that can allow management server to ssh into systemvms and VRs using proxy/jump:

• VMware →(direct agents/ms) → VR private nic

XenServer → (direct agent/ms) → ssh into XS host on port 22 → proxy jump to → VR link-local (alt. make use of XAPI vmops plugin)

• KVM → (indirect agent/ms) → KVM host/agent → ssh/forwarding → VR link-local

• Direct agents (VMware and XenServer) would ssh/scp from mgmt server to systemvms and VRs
• Indirect agents (KVM) can ssh/scp from KVM hosts to systemvms and VRs
• A hypervisor agnostic wrapper that is executed via agents to handle Cmd/Answers, but isn't aware or reduces dependency on hypervisor-specific code
• Long terms deprecate and remove router_proxy scripts
Note: this will be common architectural code that may be re-used by VR agent tooscp payload to VR/systemvms, otherwise continue to use hypervisor-specific patching/scp mechanism.

CKS Changes

Since CKS is not tied with systemvmtemplates, some optimisations can be explored:

...