Page History

...

• Use of squashfs for in-place upgrades.
• Unify initial patching via cloud-init+config-drive and deprecate/remove cmdline.
• Install/patch non-common userspace dependencies upon patching via scp/ssh (for ex. jre deb pkg etc) or serve host static repo/cache in mgmt server web server
• To build systemvmtemplate during mvn/packaging itself so hosting is no longer required
• CKS changes: since CKS is not tied with systemvmtemplates, we ca:
  • Explore: Make CKS not use cmdline string-based patching, switch to user-data or config-drive alone (Vmware disks issue); for this to happen change cloud-early-config to by default treat the template as cloud-init enable template if/when cmdline can be obtained/processed
  • Remove dockerd, docker-cli if CKS can be patched/setup using containerd alone (can reduce pkg size by 50-100MBs)

Document History

...

• Explore debootstrap, nbd and chroot based systemvm template building. PoC shows promise, builds template under 5mins. (vs 30-60mins before)
• Explore if only one of the hypervisor systemvmtemplate template can be bundled within packages it's possible for other hypervisor templates to be generated Instead of bundling all 3x systemvmtemplates, we can bundle one of the hypervisors and build/export the image for other hypervisors using qemu-img upon cloudstack-management pkg post install/upgrade step (we use qemu-img to convert qcow2 to vhd, or qcow2 to vmdk; or extra ova to convert vmdk to qcow2, vmdk to vhd/vpc).
• If building/bundling has an issue, split BO/Jenkins pkg jobs and Travis to build and host the template (for one hypervisor) alone and then bundle while packaging
• the template to different hypervisor images during build too):
  • Assume we bundle OVA/vmware; upon install/upgrade we extract the ova (tar) and grab the vmdk to export for KVM and XenServer.
  • To export for kvm, we run: qemu-img -f vmdk -O qcow2 -o compat=0.10 -c <source vmdk> <dest kvm qcow2 file>
  • To export for xenserver/xcp-ng, we run: qemu-img -f vmdk -O vpc <source vmdk> <dest xs/xcp vhd file>
  • In testing, we found this entire process of conversion takes 10-30seconds in total; which may significantly save on bandwidth of over 1G in total cloudstack-management pkg/file being downloaded
  • The logic/script that does this would check and do this kind of convertion only if qcow2 and vhds are not bundled with the cloudstack-management pkg
• During packaging, this can simplify things so if systemvm.iso is deprecated then we bundle the ova only when -D systemvm or -P systemvm is enabledFor mvn-using developers/users, the template building and bundling may require sudo access - so by default systemvmtemplate may not build let's say if -Dsystemvm is not specified.

SystemVM template upgrades

...

• Remove systemvm.iso
  • With ssh/scp setup via public ssh key of mgmt server, sent to cmdline string; we don't need systemvm.iso.
  • Deprecate and remove systemvm.iso file, move systemvmtemplate patching logic via ssh/scp.
  • Don't remove ISO drive in systemvm VM spec (so can be used in future by config drive etc)
• SSVM, CPVM CA/cert patching changes:
  • Using CA framework create certificate and pass them (pub/priv, ca cert) as key/value params in the cmdline string for all systemvms and VRs. This is do-able if we know the hostname and IP address (esp the private and link-local nic) of the systemvms/VRs.
  • The certificates can be used in future by VR agent, and currently by SSVM/CPVM agents.
  • Deprecate and remove CSR-based ssvm/cpvm certificate setup method
• Live and Automatic Patching:
  • Once ssh/scp based patching is implemented and there exists internal Java Service/APIs to trigger this; why can't we do this on the fly too for example during minor-release upgrades or upgrades where systemvmtemplate is not upgraded.
  • A new script/tool/utility to facilitate live patching (the tool is scp'd to systemvm/VR too).
  • It may also be useful to introduce a new live API or extend current API such as upgradeRouterTemplate with a new live=true|false parameter. This can allow root-admins to manually select and live-upgrade their routers without restarting network with cleanup=true.
  • Explore automatic upgrade/patching of routers, which can be triggered via some checksum-checks; or simply when restartNetwork is triggered without cleanup (i.e. VR is not destroyed) but with a new parameter to enable/allow live patching of scripts/software.
  • Live patching mechanism to explore pre/post patch and validation hooks, ability to restore upon patching failure (heal & failsafe)
  • If live patching uses ssh, can it be used to patch when upgrade older systemvms/systemvmtemplates and VR:
    • ssvm/cpvm: their internal nics, config etc hasn't changed in ages, in general, all it requires is (a) update the jars, (b) update certs incl ca-certificates, (c) updating maybe the JRE and (d) restart the cloud process
    • routers: their internal nics order hasn't changed in ages, in general it may require (a) update cloud scripts, (b) restarting all enabled services; however between major ACS versions the userspace software (haproxy, apache, dnsmasq etc) has significantly changes so those may not be guaranteed to work.
• New Multi-hop ssh based communication Utility:
  • A standard multi-hop ssh utility class/client that can allow management server to ssh into systemvms and VRs using proxy/jump:

    • VMware →(direct agents/ms) → VR private nic

    • XenServer → (direct agent/ms) → ssh into XS host on port 22 → proxy jump to → VR link-local (alt. make use of XAPI vmops plugin)

    • KVM → (indirect agent/ms) → KVM host/agent → ssh/forwarding → VR link-local

  • Direct agents (VMware and XenServer) would ssh/scp from mgmt server to systemvms and VRs
  • Indirect agents (KVM) can ssh/scp from KVM hosts to systemvms and VRs
  • A hypervisor agnostic wrapper that is executed via agents to handle Cmd/Answers, but isn't aware or reduces dependency on hypervisor-specific code
  • Long terms deprecate and remove router_proxy scripts

Misc and Good to Have Changes

...

• • Note: this will be common architectural code that may be re-used by VR agent too

CKS Changes

Since CKS is not tied with systemvmtemplates, some optimisations can be explored:

...

• Switch to using cloud as user instead of core, fix yml templatesRemove dockerd, docker-cli if CKS can be patched/setup using containerd alone; and add ssh-access in the UI if/when the user selects a SSH public key
• In the UI - display CKS host image version, dockerd and other env info that may be useful for CKS users
• Document list of support supported k8s versions with CKS with ACS release (or a test note in compat)
• Enable CKS by default
- add note in ACS release notes's compatibility page
• Optional / Good to have: Enable CKS by default (as CKS is turnkey with systemvmtemplates) and trigger registration of built-in CKS data iso per ACS version (this can similar to the built-in template, a quick to enable a k8s version with a ACS version)Add ssh-access information in UI if user-provided SSH public key