Page History

...

It will provide following features,

• Create a k8s Kubernetes cluster with desired size with different control parameters alongwith multi-master, HA support.

• Add and manage multiple Kuberneter versions with binaries ISO.

• Offline installation of k8s Kubernetes and docker binaries on cluster nodes for selected k8s Kubernetes version.

• Manage cluster, allowing it to be scaled up or down for the number of cluster nodes. Upgrade cluster to newer Kubernetes version.

• Access kube.config for cluster for accessing k8s Kubernetes dashboard.

• Delete cluster while removing nodes and k8s Kubernetes services.

Document History

References

Pull request:

Use Cases

Plugin will provide Kubernetes integration with CloudStack enabling user to run containerized services using k8s Kubernetes clusters.

Feature Specification

CloudStack Kubernetes Service plugin will add Kubernetes integration in the CloudStack. Plugin will be disabled by default and admin can control its usage using global setting. It will enable run containerized services using k8s Kubernetes clusters.

CKS will use the CoreOS based template for node VMs for running Kubernetes services. For installation of Kubernetes binaries on cluster nodes, a binaries ISO will be used to achieve faster, offline installation. Root admin will be provided with API and UI to upload/register such ISOs for a particular Kubernetes version. A shell script will be added in the codebase to easily create such ISO for a Kubernetes version.

For deployment and setup Kubernetes on cluster nodes, plugin will use Kubernetes tool, kubeadm. kubeadm is the command line tool for easily provisioning a secure Kubernetes cluster on top of physical or cloud servers or virtual machine. Under the hood, master node(s) of the cluster will start a k8s Kubernetes cluster using kubeadm init command with a custom token and worker nodes will be able to join this k8s Kubernetes cluster using kubeadm join command with the same token. More about kubeadm, https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm/

...

User will be able to create a CloudStack Kubernetes cluster using UI or API. Both UI and API will allow listing these clusters, perform lifecycle tasks over them and perform scaling for the number of nodes in a running cluster.

Minimum supported version of Kubernetes for the plugin will be 1.11

Multi-master, HA support will be available for Kubernetes version 1.16 and above.

Implementation

API

Kubernetes version management related

K8s Kubernetes version management related APIs can only be accessed by root admins

• registerK8sBinariesIsoaddKubernetesSupportedVersion

Add an a Kubernetes version that will be supported by CKS with ISO containing k8s Kubernetes binaries for a specific k8s Kubernetes version

Parameters

===========

displaytext name = (string) the display text name of the ISO. This is usually used for display purposesKubernetes supported version. (required)

name kubernetesversion = (string) the name semantic version of the ISOKubernetes. (required)

url = (string) the URL to where the ISO is currently being hosted. (required)

zoneid = (uuid) the ID of the zone you wish to register the ISO toin which Kubernetes supported version will be available.

isoid = (uuid) the ID of the binaries ISO for Kubernetes supported version.

checksum = (boolean) the MD5 checksum value of this the binaries ISO.isfeatured

url = (string) the URL of the binaries ISO for Kubernetes supported version.

• addKubernetesSupportedVersion

Delete a supported Kuberneted version

Parameters

===========

id = (uuid) the ID of the Kubernetes supported version.

deleteiso boolean) true if you want this ISO to be featured.ispublic = (boolean) true if you want to register the ISO to be publicly available to all users, false otherwise.ISO associated with the Kubernetes version to be deleted else false. Default is false.

Kubernetes supported version management related APIs can be accessed by all users

• listKubernetesSupportedVersions
  List Kubernetes supported versions
  
  Parameters
  ============
  id = (uuid) the ID of the Kubernetes supported version

Kubernetes K8s cluster management related APIs can be accessed by all users

• listKubernetesClusterlistK8sCluster
  List k8s Kubernetes cluster(s)
  
  Parameters
  ============
  id = (uuid) the ID of the container cluster
  state = (string) state of the container cluster
  name = (string) name of the container cluster
  
  
  
• createKubernetesClustercreateK8sCluster
  Creates a cluster of VM's for launching k8s Kubernetes containers.
  
  Parameters
  ============

  name = (string) name for the container cluster (required)

  description = (string) description for the container cluster

  zoneid = (uuid) availability zone in which container cluster to be launched (required)

  kubernetesversionid = (uuid) the ID of the Kubernetes version with which cluster to be created (required)

  serviceofferingid = (string) the ID of the service offering for the virtual machines in the cluster . (required)

  accountid = (uuid) an optional account for the virtual machine.

  projectid = (uuid) deploy cluster for the project

  networkid = (uuid) network in which container cluster is to be launched

  sshkeypairname = (string) name of the ssh key pair used to login to the virtual machines sshkeypair

  masternodes = (integer) number of Kubernetes cluster master nodes when setting up multi-master (default: 1)

  externalloadbalanceripaddress = (string) IP address of an exter load balancer when using multi-master on a shared network

  size = (integer) number of container cluster nodes (required)

  noderootdisksize = (long) root disk size of root disk for each node

• deleteKubernetesClusterdeleteK8sCluster
  Deletes a k8s Kubernetes cluster.
  
  Parameters
  ============
  id = (uuid) the ID of the container cluster
  
  
  

• startKubernetesClusterstartK8sCluster
  Starts a k8s Kubernetes cluster.
  
  Parameters
  ============
  id = (uuid) the ID of the container cluster
  
  
  

• stopKubernetesClusterstopK8sCluster
  Stops a k8s Kubernetes cluster.
  
  Parameters
  ============
  id = (uuid) the ID of the container cluster
  
  
  

• getKubernetesClusterConfig
  Returns kube-config file data of a Kubernetes cluster.
  
  Parameters
  ============
  id = (uuid) the ID of the Kubernetes cluster
  
  

• scaleKubernetesClustergetK8sClusterConfig
  Returns kube-config file data of a k8s cluster.
  Kubernetes cluster.
  
  Parameters
  ============
  id = (uuid) the ID of the container cluster

  size = (integer) the size or worker node count of the Kubernetes cluster

  serviceofferingid = (uuid) the ID of the service offering for the Kubernetes cluster

• upgradeKubernetesCluster
  Upgrade Kubernetes cluster to given newer version
  
  Parameters
  ============
  id = (uuid) the ID of the container cluster

  kubernetesversionid = (uuid) the ID of the Kubernetes version

Schema

1. Table for CKS cluster
   ckskubernetes_cluster:
   id (bigint, auto_increment),
   uuid (varchar, unique),
   name (varchar, unique),
   description (varchar),
   zone_id (bigint),
   service_offering_id (bigint),
   template_id (bigint),
   network_id (bigint),
   node_count (bigint),
   account_id (bigint),
   domain_id (bigint),
   state (enum/string),
   key_pair (varchar),
   cores (bigint),
   memory (bigint),
   endpoint (varchar),
   console_endpoint (varchar),
   created (datetime)
   removed  (datetime)
   
   
   

2. Table for CKS cluster details
   ckskubernetes_cluster_details:
   id (bigint, auto_increment),
   cluster_id (bigint),
   name (varchar),
   value (varchar)
   
   
   

3. Table for CKS cluster VM reference
   ckskubernetes_cluster_vm_ref:
   id (bigint, auto_increment),
   cluster_id (bigint),
   vm_id (bigint)
   
   
   

4. Table for CKS Kubernetes version
   kubernetes_supported_version:

   id (bigint, auto_increment),

   uuid (varchar, unique),

   name (varchar),

   kubernetes_verion (varchar),

   name (varchar),

   iso_id (bigint),

   zone_id (bigint),

   created (date)

   removed (date)

UI Changes

List of currently created clusters will be show in UI, in ui/plugins/cks.js.

UI will allow user to add, manage supported Kubernetes versions.

UI will allow user to perform lifecycle tasks on the cluster. It will show a tab based interface to provide cluster details, running VM instances, network rules and steps to access Kubernetes dashboard using kubectl tool.

Testing

Marvin tests cases

    CRUD tests:
        Add cluster, list clusters to confirm addition
        Delete cluster, list clusters to confirm deletion
        Scale cluster, list clusters to confirm scaling
    Cluster management tests:
    Start cluster
    Stop cluster
    Get cluster config

...