Page History

Table of Contents

WIP - this is work in progress

Introduction

Purpose: Further optimise SystemVM template generation, lifecycle, patching, seeding and upgrades.

With https://github.com/apache/cloudstack/pull/4329 in 4.16, significant systemvmtemplate lifecycle changes were introduced to simplify systemvmtemplate install /seed and upgrades, as well as making it cloud-init enabled for CKS use-case. In addition, the management server ssh public key is now also patched via cmdline string. Bundling of systemvmtemplates while makes the packages turnkey, users don't have to register/upgrade it manually now but it still has old/new pain-points:

...

patched via cmdline string

...

.

Any major feature/change should be done in the main branch, however, optimisations and operational improvements can be done in current LTS branch for future minor releases.

...

• Discussed in the CCC talk Next-Gen VR https://rohityadav.cloud/files/talks/ccc21-vr-agent.pdf
• VR agent design doc: Next-Gen Virtual Router and VR Agent

Future work

• Use of squashfs for in-place upgrades.
• Unify initial patching via cloud-init+config-drive and deprecate/remove cmdline.
• Install/patch non-common userspace dependencies upon patching via scp/ssh (for ex. jre deb pkg etc) or serve host static repo/cache in mgmt server web server
• To build systemvmtemplate during mvn/packaging itself so hosting is no longer required
• CKS changes: since CKS is not tied with systemvmtemplates, we ca:
  • Explore: Make CKS not use cmdline string-based patching, switch to user-data or config-drive alone (Vmware disks issue); for this to happen change cloud-early-config to by default treat the template as cloud-init enable template if/when cmdline can be obtained/processed

Document History

...

• Explore debootstrap, nbd and chroot based systemvm template building. PoC shows promise, builds template under 5mins. (vs 30-60mins before)
• Explore if only one of the hypervisor systemvmtemplate template can be bundled within packages it's possible for other hypervisor templates to be generated upon cloudstack-management pkg post install/upgrade step (use qemu-img to convert qcow2 to vhd, or qcow2 to vmdk; or extra ova to convert vmdk to qcow2, vmdk to vhd/vpc).
• If building/bundling has an issue, split BO/Jenkins pkg jobs and Travis to build and host the template (for one hypervisor) alone and then bundle while packaging
• , refer to PoC tree for export logic too.
• During packaging, this can simplify things so if systemvm.iso is deprecated then we bundle the ova only when -D systemvm or -P systemvm is enableFor mvn-using developers/users, the template building and bundling may require sudo access - so by default systemvmtemplate may not build let's say if -Dsystemvm is not specified.

SystemVM template upgrades

• Trigger auto-upgrades or installs based on ini file or some file/property specifies min. require template (compare against global setting and the pkg specific file/property). Same can be hard-coded/specified/used by mvn, so it doesn't auto-trigger upgrades when upgrades may not be necessary, esp for minor releases (say upgrade from 4.16.0->4.16.1...).
• Good to have: Old systemvmtemplates should be allowed to be removed/deleted by the admin

Patching Changes

• New Multi-hop ssh based Communication Utility:
  • A standard multi-hop ssh utility class/client that can allow management server to ssh into systemvms and VRs using proxy/jump:

    • VMware →(direct agents/ms) → VR private nic

    • XenServer → (direct agent/ms) → ssh into XS host on port 22 → proxy jump to → VR link-local (alt. make use of XAPI vmops plugin)

    • KVM → (indirect agent/ms) → KVM host/agent → proxy jump → VR link-local

  • Direct agents (VMware and XenServer) would ssh/scp from mgmt server to systemvms and VRs
  • Indirect agents (KVM) can ssh/scp from KVM hosts to systemvms and VRs
  • A hypervisor agnostic wrapper that is executed via agents to handle Cmd/Answers, but isn't aware or reduces dependency on hypervisor-specific code
  • Long terms deprecate and remove router_proxy scripts
• Remove systemvm.iso
  • With ssh/scp setup via public ssh key of mgmt server, sent to cmdline string; we don't need systemvm.iso.
  •  Deprecate and remove systemvm.iso file, move systemvmtemplate patching logic via ssh/scp.
  • Don't remove ISO drive in systemvm VM spec (so can be used in future by config drive etc)
• SSVM, CPVM CA/cert patching changes:
  • Using CA framework create certificate and pass them (pub/priv, ca cert) as key/value params in the cmdline string for all systemvms and VRs. This is do-able if we know the hostname and IP address (esp the private and link-local nic) of the systemvms/VRs.
  • The certificates can be used in future by VR agent, and currently by SSVM/CPVM agents.
  • Deprecate and remove CSR-based ssvm/cpvm certificate setup method
• Live and Automatic Patching:
  • Once ssh/scp based patching is implemented and there exists internal Java Service/APIs to trigger this; why can't we do this on the fly too for example during minor-release upgrades or upgrades where systemvmtemplate is not upgraded.
  • A new script/tool/utility to facilitate live patching (the tool is scp'd to systemvm/VR too).
  • It may also be useful to introduce a new live API or extend current API such as upgradeRouterTemplate with a new live=true|false parameter. This can allow root-admins to manually select and live-upgrade their routers without restarting network with cleanup=true.
  • Explore automatic upgrade/patching of routers, which can be triggered via some checksum-checks; or simply when restartNetwork is triggered without cleanup (i.e. VR is not destroyed) but with a new parameter to enable/allow live patching of scripts/software.
  • Live patching mechanism to explore pre/post patch and validation hooks, ability to restore upon patching failure (heal & failsafe)
  • If live patching uses ssh, can it be used to patch when upgrade older systemvms and VR:
    • ssvm/cpvm: their internal nics, config etc hasn't changed in ages, in general, all it requires is (a) update the jars, (b) update certs incl ca-certificates, (c) updating maybe the JRE and (d) restart the cloud process
    • routers: their internal nics order hasn't changed in ages, in general it may require (a) update cloud scripts, (b) restarting all enabled services; however between major ACS versions the userspace software (haproxy, apache, dnsmasq etc) has significantly changes so those may not be guaranteed to work.

Misc and Good to Have Changes

CKS Changes

Since CKS is not tied with systemvmtemplates, some optimisations can be explored:

• Switch to using cloud as user instead of core, fix yml templates; and add ssh-access in the UI if/when the user selects a SSH public key
• In the UI - display CKS host image version, dockerd and other env info that may be useful for CKS users
• Document list of supported k8s versions with CKS with ACS release - add note in ACS release notes's compatibility page
• Optional / Good to have: Enable CKS by default (as CKS is turnkey with systemvmtemplates) and trigger registration of built-in CKS data iso per ACS version (this can similar to the built-in template, a quick to enable a k8s version with a ACS version)
• Cloud-init changes as built-in template:
  • We've already used the systemvmtemplate now as CKS template with cloud-init setup, explore if this can be used as a cloud-init enabled built-in template
  • Use root or cloud  as default user for cloud-init
  • Migration to a new built-in template can be treated as a completely new feature and should be made/done in the main branch.
• Good to have or future changes in CKS:
• Make CKS not use cmdline string-based patching, switch to user-data or config-drive alone; for this to happen change cloud-early-config to by default treat the template as cloud-init enable template if/when cmdline can be obtained/processed
• Switch to using cloud as user instead of core, fix yml templates
• Remove dockerd, docker-cli if CKS can be patched/setup using containerd alone
• Document list of support k8s versions with CKS with ACS release (or a test note in compat)
• Enable CKS by default
Add ssh-access information in UI if user-provided SSH public key