Introduction

This functionality extends the CloudStack Kubernetes Service (CKS) in the following areas:

• Ability to specify different compute or service offerings for different types of CKS cluster nodes – worker, master or etcd:
  The createKubernetesCluster API and the corresponding UI must provide an option to provide different offering for different types of nodes. CKS compute offerings will be marked as CKS compatible.
  

• Ability to use CKS ready custom templates for CKS cluster nodes:
  CKS will allow users to specify their own templates for different CKS node types (control and worker) at the point of cluster creation. Those templates will be marked as CKS compatible.

• Ability to use generic (non CKS ready) custom templates for CKS cluster nodes:
  CKS will allow users to specify their own templates for different CKS node types (control and worker) at the point of cluster creation. Those templates will be marked as CKS compatible. The user will be responsible for installing all necessary packages in the template.

• Ability to add and remove a pre-created instance as a worker node to an existing CKS cluster:
  An instance (either virtual of physical) which has been built and prepared for CKS can been added to the desired CKS cluster. The instance must have all the CKS worker node packages installed.

• Ability to separate etcd from master nodes of the CKS cluster:
  End users should be provided with an option to separate etcd cluster at the time of CKS cluster creation. The user can enable such option in the UI or in the createKubernetesCluster API and specify the size of the etcd cluster. Based on the user inputs CloudStack should be able to provision such etcd nodes for the CKS cluster.

• Ability to mark CKS cluster nodes for manual only upgrade:
  An end user should be able to mark the desired compute offering (or the CKS template) for manual upgrades only. CKS cluster nodes marked for manual upgrade should be untouched during the Kubernetes version upgrade when executed using upgradeKubernetesCluster API.

• Ability to dedicate specific hosts/clusters to a specific domain for CKS cluster deployment:
  The dedicateHost/dedicateCluster APIs can be used to provide this functionality to dedicate hosts/clusters for CKS cluster deployments. During the deployment of CKS cluster node VMs they will by default be deployed in the dedicated cluster.

• Methodology for AS number management:
  Operators should be able to assign a range of AS numbers to an ACS Zone. ACS must have a method to assign an AS number to each Isolated network (or VPC tier), which can be retrieved via the UI and API.

• Methodology to use diverse CNI plugins (Calico, Cilium, etc…):
  End users should be able to deploy CKS clusters with Calico CNI. An option to specify which CNI plugin to be used for a CKS cluster must be provided in the createKubernetesClusterCmd API. The CNI configuration and setup can be registered as a managed userdata, and any configurable parameters – here, AS number, BGP Peer AS number and IP address, can be defined as variables in the userdata be set during the creation of the CKS cluster. This provides a flexible way for users to use the CNI plugin of their choice.

API Changes

The 'createKubernetesCluster' API has been extended to support the following parameters:

• 'nodeofferings': an optional map parameter to set the service offerings for worker, control or etcd nodes. If this parameter is not set, then every VM in the cluster will be deployed using the default service offering set on the serviceofferingid parameter.
  • Each item in the mapping must specify the following keys:
    • node: The type of node for the service offering. Allowed values: 'worker', 'control', 'etcd'
    • offering: The UUID of the service offering for the node
  • Example on cmk, setting the service offering with UUID WORKER_OFFERING_UUID to the worker nodes and the offering with UUID CONTROL_OFFERING_UUID to control nodes:
    createKubernetesCluster nodeofferings[0].node=worker nodeofferings[0].offering=<WORKER_OFFERING_UUID> nodeofferings[1].node=control nodeofferings[1].offering=<CONTROL_OFFERING_UUID>
• 'etcdnodes': an optional integer parameter to specify the number etcd nodes in the cluster, the default value is 0. In case the number is greater than 0, etcd nodes are separate from master nodes and are provisioned accordingly.
• 'nodetemplates': an optional map parameter to set the template to be used by worker, control or etcd nodes. If this parameter is not set, then every VM in the cluster will be deployed using the System VM template
  • Each item in the mapping must specify the following keys:
    • node: The type of node for the service offering. Allowed values: 'worker', 'control', 'etcd'
    • template: The UUID of the template for the node
  • Example on cmk, setting the template with UUID WORKER_TEMPLATE_UUID to the worker nodes and the template with UUID CONTROL_TEMPLATE_UUID to control nodes:
    createKubernetesCluster nodeotemplates[0].node=worker nodetemplates[0].template=<WORKER_TEMPLATE_UUID> nodetemplates[1].node=control nodetemplates[1].offering=<CONTROL_TEMPLATE_UUID>
• 'hypervisor': A new optional string parameter is added. When the hypervisor parameter is not set, the CKS cluster creation works as usual. When the hypervisor parameter is set, the CKS cluster nodes will be created on the selected hypervisor hosts (in case of mixed hypervisor environments)

The 'scaleKubertenesCluster' API has been extended to support the following parameter:

• 'nodeofferings': an optional map parameter to set the service offerings for worker, control or etcd nodes. If this parameter is not set, then every VM in the cluster will be deployed using the default service offering set on the serviceofferingid parameter.
  • Each item in the mapping must specify the following keys:
    • node: The type of node for the service offering. Allowed values: 'worker', 'control', 'etcd'
    • offering: The UUID of the service offering for the node

The 'registerTemplate' API has been extended to support the following parameter:

• 'forcks': an optional boolean parameter, false by default, which indicates if the template will be available for deploying CKS clusters

A new API has been introduced - ‘addNodesToKubernetesCluster’, which takes the following parameters as inputs:

• id: CKS cluster id to which the node(s) must be added to
• nodeids: comma separated list of (external) node (physical or virtual machines) IDs that need to be added as worker nodes to an existing managed Kubernetes cluster (CKS)
• mountcksisoonvr: (optional) Vmware only, false by default, if set to true it uses the CKS cluster network VR to mount the CKS ISO.
  When 'mountcksisoonvr' = false: The Kubernetes cluster version ISO is attached to the nodes to be added.
  When 'mountcksisoonvr' = true: The Kubernetes cluster version ISO is attached to the cluster network's virtual router and is served. The nodes to be added download the ISO content from the virtual router IP. 

A new API has been added - ‘removeNodesFromKubernetesCluster’, to support the removal of externally added worker nodes from a CKS cluster, which takes the following as input:

• id: CKS cluster id from which the node(s) must be removed from
• nodeids: comma separated list of (externally added) worker nodes IDs that need to be removed from a managed Kubernetes cluster (CKS)

Database Changes

The 'kubernetes_cluster' table has been extended, and the following columns have been added:

• 'control_service_offering_id': bigint unsigned column to store the service offering ID for Control Node(s)
• 'worker_service_offering_id': bigint unsigned column to store the service offering ID for Worker Node(s)
• 'etcd_service_offering_id': bigint unsigned column to store the service offering ID for etcd Node(s)
• 'etcd_node_count': bigint unsigned column to store the number of etcd nodes in the Kubernetes cluster
• 'control_template_id': bigint unsigned column to store the template ID to use on Control Node(s)
• 'worker_template_id': bigint unsigned column to store the template ID to use on Worker Node(s)
• 'etcd_template_id': bigint unsigned column to store the template ID to use on etcd Node(s)
• cni_config_id: userdata id representing the associated cni configuration
• cni_config_details: userdata details representing the values required for the cni configuration associated

The 'kubernetes_cluster_vm_map' table has been extended, the following column has been added:

• 'etcd_node': boolean column to indicate if a Kubernetes cluster VM is an etcd node
• 'external_node': tinyint unsigned column that indicates if the node was imported into the Kubernetes cluster
• 'manual_upgrade': tinyint unsigned column that indicates if the node is marked for manual upgrade and excluded from the Kubernetes Cluster upgrade operations.
• 'kubernetes_node_version': varchar(40) column to indicate the version of k8s the cluster node is on

The 'vm_template' table has been extended, the following column has been added:

• 'for_cks': boolean column to indicate if a template is a CKS Ready template

The 'user_data' table has been extended, and the following column has been added:

• for_cks: if true, the userdata represent CNI configuration meant for CKS use only