Introduction

ApplicationClusters (or AppC, pronounce appz) are an attempt to make orchestrating bigger application landscapes easier in a vanilla Apache CloudStack install.

Services like Kubernetes, Cloud Foundry, DBaaS require integration support from underlying CloudStack. This support includes Grouping Vms, Scaling, Monitoring. Rather than making changes every time to support various services in ACS, a generic framework has to be developed.

As an example

Table of Contents

Introduction

Services like Kubernetes, Cloud Foundry, DBaaS require integration support from underlying cloudstack. This support includes Grouping Vms, Scaling, Monitoring. Rather than making changes every time to support various services in ACS, a generic framework has to be developed.

Not only but predominantly container technologies are gaining quite a momentum and changing the way how application are traditionally deployed in the public and private clouds. Gaining interest in micro services based architecture is also fostering adooption adoption of container technologies. Much like how cloud orchestration platforms enable the provisioning of VM's VMs and adjacent services, container orchestration platforms like Kubernetes [3], docker swarm [1], mesos [2] are emerging to enable orchestration of containers. Container orchestration platforms typically can be run any where and be used to provision containers. A popular choice of running containers has been running them on the IAAS IaaS provisioned VM'sVMs. AWS and GCE provide native functionality to launch containers abstracting out the underlying consumption of VM'sVMs. A container orchestration platform can be provisioned on top of CloudStack using development tools, (see [6]), but they are not an out of the box solution. Given the momentum of container technologies, micro-services etc it make sense to provide a native functionality in CloudStack which is available out-of-the-box for users.

Purpose

Purpose of this document is present the functional requirements for supporting generic vm cluster service functionality in CloudStack

Glossary

Node - Vm in CloudStack

Machine cluster - a managed group of VMs in CloudStack

DBaaS - Database as a Service

IaaS - Infrastructure as a service

PaaS - Platform as a service

Functional specification

Container Cluster

Another example are DBaaS installations. These have different sets of roles then the above mentioned container services with different number of nodes in each role. Those two have usually only two roles but for instance sdn solutions might have three roles; switch-, controlplane- and configuration-machines.

Apache Cloudstack should not involve itself with how virtual machines are used, though plugins for CloudStack might be written that do configure sets of VMs for certain uses (like kubernetes in [8]). The intention of this functionality is to provide the organisation of sets of VMs with roles to be used as a single application, be it a container cluster or a database or a SDN facility.

Purpose

Purpose of this document is present the functional requirements for supporting generic vm cluster service functionality in CloudStack

Glossary

Node - Vm in CloudStack

Application cluster - a managed group of VMs in CloudStack

DBaaS - Database as a Service

IaaS - Infrastructure as a service

PaaS - Platform as a service

Functional specification

Application Cluster

CloudStack VM cluster service shall introduce the notion of application cluster. A 'application cluster' shall be first class CloudStack entity that will be a composite of existing CloudStack entities CloudStack vm cluster service shall introduce the notion of machine cluster. A 'machine cluster' shall be first class CloudStack entity that will be a composite of existing CloudStack entities like virtual machines, network, network rules etc.

The machine application cluster service shall stitch together cluster resources, and deploys . Any enhancements or plugins can call it to do further deploys of the chosen cluster manager like application like a manager and nodes in Kubernetes, Mesos, docker swarm etc, to provide the managers manager's service type, like AWS ECS, Google container service etc to the CloudStack users.

Cluster life-cycle management

Container service shall provide following container cluster life-cycle operations. 

• create machine application cluster: provision cluster resources, and brings the cluster in to operational readiness state. Resources provisioned shall depend on provisioning shall be the responsibility of the caller, that can act according to the cluster manager used. All the cluster VM's shall be launched in to a dedicated network for the cluster. API end point of cluster manager shall can be exposed by the caller through creating a port forwarding rule on source nat ip of the network dedicated for the cluster.
• delete machine application cluster: destroy all the resources provisioned for the machine application cluster. Post delete, a machine application cluster can not be performed any operations on it.
• start machine application cluster: Starting a cluster will start the VM's VMs and possibly start the network.
• stop machine application cluster: Stopping a cluster will shutdown all the resources consumed by the machine application cluster. User can start the cluster at a later point with Start operation.
• recovering a cluster: Due to possible faults (like VM's VMs that got stopped due to failures, or malfunctioning cluster manager etc) machine application cluster can end up in Alert state. Recover is used to revive machine application cluster to a sane running state. In the initial version this is just trying to have the correct number of VMs per role. In later versions callbacks for (re-)provisioning may be added.
• cluster resizing (scale-in/out): increase or decrease the size of the cluster on a per role basis. The functionality here is adhering to the same limitations as stated above under recovering.
• list application clusterlist machine cluster: list all the machine application clusters

provisioning service orchestrator

As The provisioning of the service is out of scope for the application cluster. A calling plugin or external tool add value by calling, as part of its creation , machine cluster shall be responsible for plan, any setting up of a control plane of the service type that was chosen. How a service will be setup is dependent on the chosen service type.

Life cycle operations

Each of the life cycle operation is a workflow resulting in either provisioning or deleting multiple CloudStack resources. There is no guarantee a workflow of a life cycle operation will succeed due to the lack of 2PC like model a two-phase-commit model, by means of resource reservation followed by provisioning semantics. Also there is no guarantee of a rollback getting succeededsucceeding. For e.g. instance, while provisioning a cluster of size 10 VM'sVMs, deployment may run out of capacity to provision any more VM's VMs after provisioning 5 Vm's the first five Vms. In which case as rollback provisioned VM's action, the provisioned VMs can be destroyed. But there can be cases where deleting a provisioned VM is not possible temporarily like disconnected hosts . For instance when a host is disconnected etc. So its not possible to achieve strong consistency and this will not be a focus in this phase of the development.

Below approach is followed while performing life cycle operations..

• A best effort will be done to bring the cluster up to spec. If this failed it will be retried indefinitely.
• If deployment fails it is the responsibility of the user to stop and destroy the cluster.
• Do a best effort rollback for a life cycle operation in case of failure
• In case rollback fails, have reconciliation mechanisms that will ensure eventual consistency

The below state machine reflects how a machine application cluster state transitions for each of life cycle operations

Image Removed

CREATE TABLE IF NOT EXISTS `cloud`.`application_cluster` (
    `id` bigint unsigned NOT NULL auto_increment COMMENT 'id',
    `uuid` varchar(40),
    `name` varchar(255) NOT NULL,
    `description` varchar(4096) COMMENT 'display text for this application cluster',
    `zone_id` bigint unsigned NOT NULL COMMENT 'zone id',
    `network_id` bigint unsigned COMMENT 'network this application cluster uses',
    `account_id` bigint unsigned NOT NULL COMMENT 'owner of this cluster',
    `domain_id` bigint unsigned NOT NULL COMMENT 'owner of this cluster',
    `state` char(32) NOT NULL COMMENT 'current state of this cluster',
    `key_pair` varchar(40),
    `created` datetime NOT NULL COMMENT 'date created',
    `removed` datetime COMMENT 'date removed if not null',
    `gc` tinyint unsigned NOT NULL DEFAULT 1 COMMENT 'gc this application cluster or not',
    `network_cleanup` tinyintCREATE TABLE IF NOT EXISTS `cloud`.`machine_cluster` (
    `id` bigint unsigned NOT NULL auto_incrementDEFAULT 1 COMMENT 'id',
    `uuid` varchar(40),
    `name` varchar(255) NOT NULL,
    `description` varchar(4096) COMMENT 'display text for this machine cluster',
    `zone_id` bigint unsigned NOT NULL COMMENT 'zone id',
    `service_offering_id` bigint unsigned COMMENT 'service offering id for the cluster VM',
    `template_id` bigint unsigned COMMENT 'vm_template.id',
    `network_id` bigint unsigned COMMENT 'network this machine cluster uses',
    `node_count` bigint NOT NULL default '0true if network needs to be clean up on deletion of application cluster. Should be false if user specfied network for the cluster',
    CONSTRAINT `fk_cluster__zone_id` FOREIGN KEY `fk_cluster__zone_id` (`zone_id`) REFERENCES `data_center` (`id`) ON DELETE CASCADE,
    CONSTRAINT `fk_cluster__network_id` FOREIGN KEY `fk_cluster__network_id`(`network_id`) REFERENCES `networks`(`id`) ON DELETE CASCADE,
    PRIMARY KEY(`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

 
CREATE TABLE IF NOT EXISTS `cloud`.`application_cluster_role` (
    `id` bigint unsigned NOT NULL auto_increment COMMENT 'id',
    `account`cluster_id` bigint unsigned NOT NULL COMMENT 'owner of this clustercluster id',
    `domain_id` bigint unsigned`name` varchar(255) NOT NULL COMMENT 'owner of this clusterrole name',
    `service_offering_id` `state` char(32) NOT NULL bigint unsigned COMMENT 'currentservice offering stateid offor thisthe cluster VM',
    `key_pair` varchar(40)`template_id` bigint unsigned COMMENT 'vm_template.id',
    `cores``node_count` bigint unsigned NOT NULL COMMENTdefault 'number of cores0',
     `memory` bigint unsigned NOT NULL COMMENT 'total memory',
    `endpoint` varchar(255) COMMENT 'url endpoint of the machine cluster manager api access',
    `console_endpoint` varchar(255) COMMENT 'url for the machine cluster manager dashbaord',
    `created` datetime NOT NULL COMMENT 'date created',
    `removed` datetime COMMENT 'date removed if not null',
    `gc` tinyintPRIMARY KEY(`id`),
    CONSTRAINT `fk_cluster__service_offering_id` FOREIGN KEY `fk_cluster__service_offering_id` (`service_offering_id`) REFERENCES `service_offering`(`id`) ON DELETE CASCADE,
    CONSTRAINT `fk_cluster__template_id` FOREIGN KEY `fk_cluster__template_id`(`template_id`) REFERENCES `vm_template`(`id`) ON DELETE CASCADE,
    CONSTRAINT `application_cluster_role_cluster__id` FOREIGN KEY `application_cluster_role_cluster__id`(`cluster_id`) REFERENCES `application_cluster`(`id`) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

 
CREATE TABLE IF NOT EXISTS `cloud`.`application_cluster_role_vm_map` (
    `id` bigint unsigned NOT NULL DEFAULTauto_increment 1 COMMENT 'gc this machine cluster or notid',
    CONSTRAINT `fk_cluster__zone_`role_id` FOREIGNbigint KEY `fk_cluster__zone_id` (`zone_id`) REFERENCES `data_center` (`id`) ON DELETE CASCADEunsigned NOT NULL COMMENT 'role id',
    CONSTRAINT `fk_cluster__service_offering_id` FOREIGN KEY `fk_cluster__service_offering_id` (`service_offering_id`) REFERENCES `service_offering`(`id`) ON DELETE CASCADE`vm_id` bigint unsigned NOT NULL COMMENT 'vm id',
    PRIMARY KEY(`id`),
    CONSTRAINT `fk`application_cluster_role_template_id` FOREIGN KEY `fkvm_map_cluster_role_template_id`(`template_id`) REFERENCES `vm_template`(`id`) ON DELETE CASCADE,
    CONSTRAINT `fkFOREIGN KEY `application_cluster_role_network_id` FOREIGN KEY `fkvm_map_cluster_role_network_id`(`network`role_id`) REFERENCES `networks``application_cluster_role`(`id`) ON DELETE CASCADE,
    PRIMARY KEY(`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

 
CREATE TABLE IF NOT EXISTS `cloud`.`machine`application_cluster_vm_map`details` (
    `id` bigint unsigned NOT NULL auto_increment COMMENT 'id',
    `cluster_id` bigint unsigned NOT NULL COMMENT 'cluster id',
    `vm_id` bigint unsigned`key` varchar(255) NOT NULL,
   COMMENT 'vm`value` id'text,
    PRIMARY KEY(`id`),
    CONSTRAINT `machine`application_cluster_vm_mapdetails_cluster__id` FOREIGN KEY `machine`application_cluster_vmdetails_map_cluster__id`(`cluster_id`) REFERENCES `machine`application_cluster`(`id`) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

 
CREATE TABLE IF NOT EXISTS `cloud`.`machine`application_cluster_role_details` (
    `id` bigint unsigned NOT NULL auto_increment COMMENT 'id',
    `cluster`role_id` bigint unsigned NOT NULL COMMENT 'clusterrole id',
    `username``key` varchar(255) NOT NULL,
    `password` varchar(255) NOT NULL,
    `registry_username` varchar(255),
    `registry_password` varchar(255),
    `registry_url` varchar(255),
    `registry_email` varchar(255),
    `network_cleanup` tinyint unsigned NOT NULL DEFAULT 1 COMMENT 'true if network needs to be clean up on deletion of machine cluster. Should be false if user specfied network for the cluster',
    PRIMARY KEY(`id`),
    CONSTRAINT `machine_cluster_details_cluster__id` FOREIGN KEY `machine_cluster_details_cluster__id`(`cluster_id`) REFERENCES `machine_cluster`(`id`) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8`value` text,
    PRIMARY KEY(`id`),
    CONSTRAINT `application_cluster_role_details_role__id` FOREIGN KEY `application_cluster_role_details_cluster__id`(`role_id`) REFERENCES `application_cluster_role`(`id`) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

// example details for  a cluster used as a k8 container cluster:
enum {
  `username`,
  `password`,
  `registry_username`,
  `registry_password`,
  `registry_url`,
  `registry_email`,
  `endpoint` varchar(255) COMMENT 'url endpoint of the application cluster manager api access',
  `console_endpoint` varchar(255) COMMENT 'url for the application cluster manager dashbaord',
  `cores` bigint unsigned NOT NULL COMMENT 'number of cores',
  `memory` bigint unsigned NOT NULL COMMENT 'total memory'
};

References 

[1] https://www.docker.com/products/docker-swarm

...

[7] https://github.com/kubernetes/kubernetes/tree/master/cluster/rackspace/cloud-config

[8] https://github.com/shapeblue/ccs