Info | ||
---|---|---|
| ||
This design, though valid is ignoring the rising use of tools like oasis camp and tosca, as well as the more propriatary format of terraform. Embedding or co-installing Apache Brooklyn with CloudStack for the use of creating application landscapes seems more appropriate. |
Table of Contents |
---|
ApplicationClusters (or AppC, pronounce appz) are an attempt to make orchestrating bigger application landscapes easier in a vanilla Apache CloudStack install.
Services like Kubernetes, Cloud Foundry, DBaaS require integration support from underlying CloudStack. This support includes Grouping Vms, Scaling, Monitoring. Rather than making changes every time to support various services in ACS, a generic framework has to be developed.
As an example container Container technologies are gaining quite a momentum and changing the way how application are traditionally deployed in the public and private clouds. Gaining interest in micro services based architecture is also fostering adaption adoption of container technologies. Like Much like how cloud orchestration platforms enabled enable the provisioning of VM's VMs and adjunct adjacent services, container orchestration platforms like Kubernetes [3], docker swarm [1], mesos [2] are emerging to enable orchestration of containers. Container orchestration platforms typically can be run any where and can be used to provision containers. A popular choice of running containers has been running them on the IAAS IaaS provisioned VM'sVMs. AWS and GCE provides provide native functionality to launch containers abstracting out the underlying consumption of VM's. There are couple efforts to provision a container orchestration platforms VMs. A container orchestration platform can be provisioned on top of CloudStack using development tools, (see [6]), but they are not an out of the box solution. Given the momentum of container technologies, miromicro-services etc it make sense to provide a native functionality in CloudStack which is available out-of-the-box for users.
Another example are DBaaS installations. These have different sets of roles then the above mentioned container services with different number of nodes in each role. Those two have usually only two roles but for instance sdn solutions might have three roles; switch-, controlplane- and configuration-machines.
Apache Cloudstack should not involve itself with how virtual machines are used, though plugins for CloudStack might be written that do configure sets of VMs for certain uses (like kubernetes in [8]). The intention of this functionality is to provide the organisation of sets of VMs with roles to be used as a single application, be it a container cluster or a database or a SDN facility.
Purpose of this document is present the functional requirements for supporting native generic vm cluster service functionality in CloudStack to provision containers and detail design aspects of how the functionality will be achieved.
Node - Vm in CloudStack
Application cluster - a managed group of VMs in CloudStack
DBaaS - Database as a Service
IaaS - Infrastructure as a service
PaaS - Platform as a service
...
CloudStack container VM cluster service shall introduce the notion of container application cluster. A 'container application cluster' shall be first class CloudStack entity that will be a composite of existing CloudStack entities like virtual machines, network, network rules etc. Container
The application cluster service shall stitch together container cluster resources, and deploys chosen cluster manager like . Any enhancements or plugins can call it to do further deploys of the chosen cluster application like a manager and nodes in Kubernetes, Mesos, docker swarm etc, to provide a container service the manager's service type, like AWS ECS, Google container service etc to the CloudStack users.
Container service shall provide following container cluster life-cycle operations.
...
As part of container cluster creation, container service shall be responsible for setting up control place of container orchestrator that was choosen.
The provisioning of the service is out of scope for the application cluster. A calling plugin or external tool add value by calling, as part of its creation plan, any setting up of a control plane of the service type that was chosen. How a service will be setup is dependent on the chosen service type.
Following API shall be introduced with application cluster:
New reponse 'applicationclusterreponse' shall be added with below details:
...
Each of the life cycle operation is a workflow resulting in either provisioning or deleting multiple CloudStack resources. Its not possible to achieve atomicity. There is no guarantee a workflow of a life cycle operation will succeed due to the lack of 2PC like model a two-phase-commit model, by means of resource reservation followed by provisioning semantics. Also there is no guarantee of a rollback getting succeededsucceeding. For e.g. instance, while provisioning a cluster of size 10 VM'sVMs, deployment may run out of capacity to provision any more VM's VMs after provisioning 5 Vm's the first five Vms. In which case as rollback provisioned VM's action, the provisioned VMs can be destroyed. But there can be cases where deleting a provisioned VM is not possible temporarily like disconnected hosts etc. . For instance when a host is disconnected etc. So its not possible to achieve strong consistency and this will not be a focus in this phase of the development.
Below approach is followed while performing life cycle operations..
Below state machine captures the state of container cluster as it goes through various life-cycle operations. Not all states are necessarily end user visible.
The below state machine reflects how a application cluster state transitions for each of life cycle operations
Gliffy Diagram | ||||
---|---|---|---|---|
|
Garbage collection shall
garbage collection will be implemented as a way to clean up the resources of container application cluster, as a background task. Following are cases where cluster resources are freed up.
If there is are failures in cleaning up resources, and clean up can not proceed, the state of container the application cluster is marked in as 'Expunge' state from instead of 'Expunging' state. Garbage The garbage collector will loop through the list of container application clusters in 'Expunge' state periodically and try to free the resources held by container cluster.
should we care to implement rollback of failure in container cluster creation, or do a lazy cleanup. Which is to mark the container cluster to be in 'Expunging' state and let garbage collector do the cleanup. Its just matter of when to do it. Both the flows may be using same cleanup moduleapplication cluster.
State of the container application cluster is 'desired state' of the cluster as intended by the user or what the system's logical view of the container application cluster. However there are various scenarios where desired state of the container application cluster is not sync with state that can be inferred from actual physical/infrastructure. For e.g a container application cluster in 'Running' state with cluster size of 10 VM's all in running state. Its possible due to host failures, some of the VM's may get stopped at later point. Now the desired state of the container application cluster is a cluster with 10 VM's s running and in operationally ready state (w.r.t to container provisioning), but the resource layer is state is different. So we need a mechanism to ensure:
Following mechanism will be implemented.
State transitions in FSM, where a container application cluster ends up in 'Alert' state:
From layering perspective, CCS is like layered on top of CloudStack functionality. There is no way to control the life-cycle of individual resources that are part of container cluster. For e.g user can go and delete the VM's that are part of container cluster.
OPEN QUESTION There are no hooks to restrict this actions?
Only design option is to cluster state synchronization to figure missing entities (in case of destroyed VM's) or conflicting states (User can stop a VM, that is expected to be running by CCS) and put the cluster state in alert.
Policies can be defined on how to recover the cluster.
re-use cloud DB vs keep separate DB
Core OS template shall be used to provision container cluster VM. Setting up a cluster VM as master/node of kubernetes is done through cloud-config script [7] in CoreOS. CloudStack shall pass necessary cloud config script as base 64 encoded user data. Once Core OS instances are launched by CloudStack, by virtue of cloud-config data passed as user data, core OS instances self-configures as kubernetes master and node VM's
Code Block | ||
---|---|---|
| ||
CREATE TABLE IF NOT EXISTS `cloud`.`application_cluster` (
`id` bigint unsigned NOT NULL auto_increment COMMENT 'id',
`uuid` varchar(40),
`name` varchar(255) NOT NULL,
`description` varchar(4096) COMMENT 'display text for this application cluster',
`zone_id` bigint unsigned NOT NULL COMMENT 'zone id',
`network_id` bigint unsigned COMMENT 'network this application cluster uses',
`account_id` bigint unsigned NOT NULL COMMENT 'owner of this cluster',
`domain_id` bigint unsigned NOT NULL COMMENT 'owner of this cluster',
`state` char(32) NOT NULL COMMENT 'current state of this cluster',
`key_pair` varchar(40),
`created` datetime NOT NULL COMMENT 'date created',
`removed` datetime COMMENT 'date removed if not null',
`gc` tinyint unsigned NOT NULL DEFAULT 1 COMMENT 'gc this application cluster or not',
`network_cleanup` tinyint unsigned NOT NULL DEFAULT 1 COMMENT 'true if network needs to be clean up on deletion of application cluster. Should be false if user specfied network for the cluster',
CONSTRAINT |
Handling out-of-band changes:
CCS will keep below book keeping tables to store the cloudstack resources provisioned and used for a container cluster.
Note there are no foreign key and delete cascades. CCS should not loose book keeping data on the resources even if resource is deleted from the CloudStack DB.
CCS code need to do defensive coding to verify entity exist in CloudStack tables before using it.
CREATE
TABLE
IF
NOT
EXISTS `cloud`.`container_cluster` (
`id`
bigint
unsigned
NOT
NULL
auto_increment COMMENT
'id'
,
`uuid`
varchar
(40),
`
name
`
varchar
(255)
NOT
NULL
,
`description`
varchar
(4096)
NULL
COMMENT
'description'
,
`zone_id`
bigint
unsigned
NOT
NULL
COMMENT
'zone id'
,
`service_offering_id`
bigint
unsigned COMMENT
'service offering id for the cluster VM'
,
`network_id`
bigint
unsigned COMMENT
'network this public ip address is associated with'
,
`node_count`
bigint
NOT
NULL
default
'0'
,
`account_id`
bigint
unsigned
NOT
NULL
COMMENT
'owner of this cluster'
,
`domain_id`
bigint
unsigned
NOT
NULL
COMMENT
'owner of this cluster'
,
`state`
char
(32)
NOT
NULL
COMMENT
'current state of this cluster'
,
`key_pair`
varchar
(40),
`cores`
bigint
unsigned
NOT
NULL
COMMENT
'number of cores'
,
`memory`
bigint
unsigned
NOT
NULL
COMMENT
'total memory'
,
`endpoint`
varchar
(255) COMMENT
'url endpoint of the container cluster manager api access'
,
`console_endpoint`
varchar
(255) COMMENT
'url for the container cluster manager dashbaord'
,
`fk_cluster__zone_id` FOREIGN KEY `fk_cluster__zone_id` (`zone_id`) REFERENCES `data_center` (`id`) ON DELETE CASCADE |
, CONSTRAINT `fk_cluster__network_id` FOREIGN KEY `fk_cluster__network_id`(`network_id`) REFERENCES `networks`(`id`) ON DELETE CASCADE, PRIMARY KEY(`id`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8; CREATE TABLE IF NOT EXISTS `cloud`.`application_cluster_role` ( `id` bigint unsigned NOT NULL auto_increment COMMENT 'id', `cluster_id` bigint unsigned NOT NULL COMMENT 'cluster id', `name` varchar(255) NOT NULL COMMENT 'role name', `service_offering_id` bigint unsigned COMMENT 'service offering id for the cluster VM', `template_id` bigint unsigned COMMENT 'vm_template.id', `node_count` bigint NOT NULL default '0', PRIMARY KEY(`id`), CONSTRAINT `fk_cluster__service_offering_id` FOREIGN KEY `fk_cluster__service_offering_id` (`service_offering_id`) REFERENCES `service_offering`(`id`) ON DELETE CASCADE, |
CONSTRAINT `fk_cluster__template_id` FOREIGN KEY `fk_cluster__template_id`(`template_id`) REFERENCES `vm_template`(`id`) ON DELETE CASCADE, |
CONSTRAINT `application_cluster_role_ |
cluster__id` FOREIGN KEY |
`application_cluster_role_ |
cluster__id`( |
`cluster_id`) REFERENCES |
`application_cluster`(`id`) ON DELETE |
PRIMARY
KEY
(`id`)
) ENGINE=InnoDB
DEFAULT
CASCADE ) ENGINE=InnoDB DEFAULT CHARSET=utf8; |
CREATE
TABLE
IF
NOT
EXISTS `cloud`.`containerCREATE TABLE IF NOT EXISTS `cloud`.`application_cluster_role_vm_map` |
`id`
bigint
unsigned
NOT
NULL
auto_increment COMMENT
'id'
,
`cluster_id`
bigint
unsigned
NOT
NULL
COMMENT
'cluster id'
,
`vm_id`
bigint
unsigned
NOT
NULL
COMMENT
'vm id'
,
PRIMARY
KEY
(`id`),
-- CONSTRAINT `container_cluster_vm_map_cluster__id` FOREIGN KEY `container_cluster_vm_map_cluster__id`(`cluster_id`) REFERENCES
`container_cluster`(`id`)
ON
DELETE
CASCADE
) ENGINE=InnoDB
DEFAULT
CHARSET=utf8;
...
(
`id` bigint unsigned NOT NULL auto_increment COMMENT 'id',
`role_id` bigint unsigned NOT NULL COMMENT 'role id',
`vm_id` bigint unsigned NOT NULL COMMENT 'vm id',
PRIMARY KEY(`id`),
CONSTRAINT `application_cluster_role_vm_map_cluster_role__id` FOREIGN KEY `application_cluster_role_vm_map_cluster_role__id`(`role_id`) REFERENCES `application_cluster_role`(`id`) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
CREATE TABLE IF NOT EXISTS `cloud`.`application_cluster_details` (
`id` bigint unsigned NOT NULL auto_increment COMMENT 'id',
`cluster_id` bigint unsigned NOT NULL COMMENT 'cluster id',
`key` varchar(255) NOT NULL,
`value` text,
PRIMARY KEY(`id`),
CONSTRAINT `application_cluster_details_cluster__id` FOREIGN KEY `application_cluster_details_cluster__id`(`cluster_id`) REFERENCES `application_cluster`(`id`) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
CREATE TABLE IF NOT EXISTS `cloud`.`application_cluster_role_details` (
`id` bigint unsigned NOT NULL auto_increment COMMENT 'id',
`role_id` bigint unsigned NOT NULL COMMENT 'role id',
`key` varchar(255) NOT NULL,
`value` text,
PRIMARY KEY(`id`),
CONSTRAINT `application_cluster_role_details_role__id` FOREIGN KEY `application_cluster_role_details_cluster__id`(`role_id`) REFERENCES `application_cluster_role`(`id`) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8; |
Code Block | ||
---|---|---|
| ||
// example details for a cluster used as a k8 container cluster:
enum {
`username`,
`password`,
`registry_username`,
`registry_password`,
`registry_url`,
`registry_email`,
`endpoint` varchar(255) COMMENT 'url endpoint of the application cluster manager api access',
`console_endpoint` varchar(255) COMMENT 'url for the application cluster manager dashbaord',
`cores` bigint unsigned NOT NULL COMMENT 'number of cores',
`memory` bigint unsigned NOT NULL COMMENT 'total memory'
}; |
[1] https://www.docker.com/products/docker-swarm
[2] https://mesosphere.github.io/marathon/
[4] https://aws.amazon.com/ecs/
[5] https://cloud.google.com/container-engine/
[6] https://cloudierthanthou.wordpress.com/2015/10/23/apache-mesos-and-kubernetes-on-apache-cloudstack/
[7] https://github.com/kubernetes/kubernetes/tree/master/cluster/rackspace/cloud-config