This functionality extends the CloudStack Kubernetes Service (CKS) in the following areas:
Ability to specify different compute or service offerings for different types of CKS cluster nodes – worker, master or etcd:
The createKubernetesCluster API and the corresponding UI must provide an option to provide different offering for different types of nodes. CKS compute offerings will be marked as CKS compatible.
Ability to use CKS ready custom templates for CKS cluster nodes:
CKS will allow users to specify their own templates for different CKS node types (control and worker) at the point of cluster creation. Those templates will be marked as CKS compatible.
Ability to use generic (non CKS ready) custom templates for CKS cluster nodes:
CKS will allow users to specify their own templates for different CKS node types (control and worker) at the point of cluster creation. Those templates will be marked as CKS compatible. The user will be responsible for installing all necessary packages in the template.
Ability to add and remove a pre-created instance as a worker node to an existing CKS cluster:
An instance (either virtual of physical) which has been built and prepared for CKS can been added to the desired CKS cluster. The instance must have all the CKS worker node packages installed.
Ability to separate etcd from master nodes of the CKS cluster:
End users should be provided with an option to separate etcd cluster at the time of CKS cluster creation. The user can enable such option in the UI or in the createKubernetesCluster API and specify the size of the etcd cluster. Based on the user inputs CloudStack should be able to provision such etcd nodes for the CKS cluster.
Ability to mark CKS cluster nodes for manual only upgrade:
An end user should be able to mark the desired compute offering (or the CKS template) for manual upgrades only. CKS cluster nodes marked for manual upgrade should be untouched during the Kubernetes version upgrade when executed using upgradeKubernetesCluster API.
Ability to dedicate specific hosts/clusters to a specific domain for CKS cluster deployment:
The dedicateHost/dedicateCluster APIs can be used to provide this functionality to dedicate hosts/clusters for CKS cluster deployments. During the deployment of CKS cluster node VMs they will by default be deployed in the dedicated cluster.
Methodology for AS number management:
Operators should be able to assign a range of AS numbers to an ACS Zone. ACS must have a method to assign an AS number to each Isolated network (or VPC tier), which can be retrieved via the UI and API.
Methodology to use diverse CNI plugins (Calico, Cilium, etc…):
End users should be able to deploy CKS clusters with Calico CNI. An option to specify which CNI plugin to be used for a CKS cluster must be provided in the createKubernetesClusterCmd API. The CNI configuration and setup can be registered as a managed userdata, and any configurable parameters – here, AS number, BGP Peer AS number and IP address, can be defined as variables in the userdata be set during the creation of the CKS cluster. This provides a flexible way for users to use the CNI plugin of their choice.
The 'createKubernetesCluster' API has been extended to support the following parameters:
'nodeofferings': an optional map parameter to set the service offerings for worker, control or etcd nodes. If this parameter is not set, then every VM in the cluster will be deployed using the default service offering set on the serviceofferingid parameter.
Each item in the mapping must specify the following keys:
node: The type of node for the service offering. Allowed values: 'worker', 'control', 'etcd'
offering: The UUID of the service offering for the node
Example on cmk, setting the service offering with UUID WORKER_OFFERING_UUID to the worker nodes and the offering with UUID CONTROL_OFFERING_UUID to control nodes: createKubernetesCluster nodeofferings[0].node=worker nodeofferings[0].offering=<WORKER_OFFERING_UUID> nodeofferings[1].node=control nodeofferings[1].offering=<CONTROL_OFFERING_UUID>
'etcdnodes': an optional integer parameter to specify the number etcd nodes in the cluster, the default value is 0. In case the number is greater than 0, etcd nodes are separate from master nodes and are provisioned accordingly.
'nodetemplates': an optional map parameter to set the template to be used by worker, control or etcd nodes. If this parameter is not set, then every VM in the cluster will be deployed using the System VM template
Each item in the mapping must specify the following keys:
node: The type of node for the service offering. Allowed values: 'worker', 'control', 'etcd'
template: The UUID of the template for the node
Example on cmk, setting the template with UUID WORKER_TEMPLATE_UUID to the worker nodes and the template with UUID CONTROL_TEMPLATE_UUID to control nodes: createKubernetesCluster nodeotemplates[0].node=worker nodetemplates[0].template=<WORKER_TEMPLATE_UUID> nodetemplates[1].node=control nodetemplates[1].offering=<CONTROL_TEMPLATE_UUID>