Page History

Bug Reference

CLOUDSTACK-241

Branch

regions

https://git-wip-us.apache.org/repos/asf?p=incubator-cloudstack.git;a=shortlog;h=refs/heads/regions

Introduction

Purpose

The objective of this feature is to add AWS EC2 like Regions implementation into CloudStack.  Regions are dispersed and located in separate geographic areas. Availability Zones(or Zones in CloudStack) are distinct locations within a Region that are engineered to be isolated from failures in other Zones and provide inexpensive, low latency network connectivity to other Zones in the same Region

Regions would provide the following benefits:

...

Requirements Doc: Regions requirements

Document History

Glossary

Feature Specifications

Introduce the notion of Regions, which will be managed independently by a separate management servers.  With Regions, infrastructure will be organized as follows:

Region -> Zone -> Pod -> Cluster

 Accounts

 User Accounts will be available across Regions. User should be able to use the same account in all Regions. Switching between Regions should not require the user to sign-on again (SSO should be supported).in again

Templates

 CS Templates are currently zone specific. Templates should be available to all zones within a region. User should be able to migrate templates from one Region to another. This requires S3 like object store

...

 An authentication service will available for components like object-store to authenticate and get account information using getUser API

Custom Authentication Adapter

Existing UserAuthenticator will be enhanced to support provisioning from directory services like LDAP and Active Directory.

Account can also be auto-provisioned using the directory service

Assumptions

Object Store: This feature assumes that S3 like object store is available. Either implemented in CloudStack or through integrationsthrough integration.

Templates, Snapshots, EIP and ELP work is not part of this spec. There are being tracked separately as per JIRA tickets mentioned above.

Account Provisioning: Typically account provisioning is external to CloudStack. Adding accounts in all regions and keeping the data in sync is not handled by CloudStack. Only events are generated when changes are made.

Usage

1. Usage records will be generated by each Region separately. Portal layer above CS should combine usage across Regions into a consolidated invoice.

Use cases

(Using external provisioning system)

• Create Account
  1. Create Account

...

• 1. in the external provisioning system with unique Id
  2. Using CS API create account in all regions using the unique Id created in step1 as UUID. CreateAccount API takes UUID as an optional parameter. When UUID is provided, CloudStack does not generate a UUID and instead will use external ID.
  3. External Id(UUID) for the Account should be same in all Regions.

...

• 1. Account

...

• 1. DB Ids could be different for each Region.

...

• Update Account (update/disable/enable Account)

...

• 1. Update account in external provisioning system
  2. Update account in all regions via API.
• Delete Account
  1. Delete

...

• 1. account

...

• 1. in external provisioning system
  2. Delete account in all regions via API

Similarly for users and domains.

...

Architecture and Design description

Database

Each Region will have a separate database. Below data will be common to all databases across Regions:

1. Account
2. User
3. DomainRegion related meta data

Remaining data is expected to be per-Region(including projects, global config, resource limits) and not shared across databases in other Regions. 

Data synchronization

Accounts:

• When creating the account, it is created first in the Region that the user is logged into
• Account table will include region_id column, indicating the Region that it is created from
• This Region is responsible for propagation of Account creation to other Regions
• Account deletion is always forwarded to the Region that originally created the account 
• The Region that originally created account is responsible for deleting the account from all of the regions before it declares the account is deleted. If it cannot reach an Region, account deletion fails.

The same applies for Domains.

Create: For any resource created in Region A, MS is Region A  will send API requests to peer Regions along with external Id of the resource. API requests are sent to the peer Regions using the end_point of each Region

New Tables

Event framework

Events are published using the event framework.Update/Remove: Request is forwarded to the source region. After updating the resource locally, source region sends API requests to peer regions with ispropogate flag set to true.

Authentication

Once logged in User should be able to switch from one Region to another without providing credentials again (Single sign-on?)

Secondary Storage

Snapshots and Templates are currently stored on secondary storage. After snapshots/templates are moved to object store, role of secondary storage will change. There were proposals also to move secondary storage to pod-level.

web services APIs

New APIs

addRegion

Registers a region into another region

Request Parameters:

1. id: Region Id(int) Region Id 
2. name: Name (String) Name of the Region
3. endpoint: Region (String) Region end point. e.g. http://10.147.30.11:8080/client
4. userapikey: API key of the Admin user
5. usersecretkey: Secret key of Admin user

updateRegion

Parameters

Response Parameters:

1. id: Region Id 
2. name: Name of the Region
3. endpoint: Region end point

updateRegion

Updates region details

Request Parameters:

1. id: (int)id: Region Id
2. name: (String)Name of the Region
3. endpoint: (String)Region end point. e.g. https://10.147.30.11:8080/client
4. userapikey: API key of the Admin user
5. usersecretkey: Secret key of Admin user

removeRegion

Parameters

Response Parameters:

1. id: Region Id 
2. name: Name of the Region
3. endpoint: Region end point

removeRegion

Removes region from current region.

Request Parameters:

1. id: (int)id: id of the Region to  be removed

Response Parameters:

1. Boolean success

listRegions

list all Regions. Can be filtered by id or name

Request Parameters:

1. id: list by Region Id
2. name: (String)list by Region Name

Response Parameters (List):

1. id: Region Id 

getUser

1. name: Name of the Region
2. endpoint: Region end point

getUser

Admin only API. Get user details by api_key. 

Request Parameters:Parameters 

1. userapikey: (String)Get user details by API key 

Changes to existing APIs

• createAccount  createAccount *# New parameters
  1. *## accountid : External Id for the Account. Could be UUID or domainpath + username or any string unique across Regionsuserid: External Id UUID of the account
     1. userid: UUID for the User
• createUser
  • New parameter
    • UUID for the User. 
• regionid: Id of the source Region where Account creation request was initiated

• createDomain deleteAccount, updateAccount, disableAccount, enableAccount
  • New
  parameters
  • parameter
    ispropagate: Boolean flag which indicated if the API call is being propagated from another Region
    • • domainid: UUID for the Domain

  Limitations

  1. Since all update/delete operations are forwarded to the source region (i.e the region where the resource was initially created), if the source region is down, these operations will failAccount/User/Domain data propogation/sync has to be handled outside cloudstack

  2. Only events will be generated by cloudstack

  Upgrade

  During upgrade, flexibility will be provided to move existing zones into any Region.

  ...

  • Each zone becomes part of a separate Region: Copy existing DB to all Regions and disable all zones except one in each Region.
    • Upgrade existing MS to 4.1. This MS will become region 1
    • Disable all the zones in region 1
    • Dump region 1 DB
      • mysqldump -u cloud -p -h <region1_db_host> cloud  > region1.sql
    • Install 4.1 MS in the remaining Regions
      • Set the region_id while installing the DB
      • cloud-setup-databases cloud:<dbpassword>@localhost --deploy-as=root:<password> -e <encryption_type> -m <management_server_key> -k <database_key> -r <region_id>
    • Copy region 1 DB to other regions*** mysql -u cloud -p -h <region2_db_host> cloud < region1.sql
    • Start all management servers. At this point all the zones are disabled
    • Selectively enable zones in the required regions
  • All zones remain in the same Region: Nothing to do here. 
  • Zones are divided among Regions:  Copy existing DB to all regions and disabling the zones in all Regions other than the selected Region

  Scripts

  Offline scripts will be provided to move data across regions to ease

  • upgrade 
  • add new region
  • remove regionFirst Region Id will become the source Region Id for all Account/User/Domain resources 

  UI flow

  • User/Admin should be able to view all Regions by logging into a MS of any of the Regions. User then should be able to select a specific Region to view details of that Region.Reports on the dashboard
  • Users should be aggregated to Region level
  • On first UI access (after MS installation), user should be able to connect to another CloudStack MS (in another Region) or treat this instance as the first region
  • Modify the start-up wizard to force users define the Region (Region Name, Description, etc) before they start creating the Availability Zones
  Users should be able to switch between various regions for UI using Single Sign-On.
  • able to switch between various regions for UI using Single Sign-On.

  Sample Workflow

  Single Region

  If an environment has only 1 region, functionality will be same as the current CS installation.  Id of this region will be 1. All accounts/users/domains will have region_id as 1

  Default local region will be added with name "Local" and end_point "http://localhost:8080/client". Use updateRegion API to set a different name or end_point for this region

  Adding 2nd Region

  1. Install a 2nd CS instance.

  2. While installing database set region_id using -r option in cloud-setup-databases script (Make sure database_key is same across all regions).

  cloud-setup-databases cloud:<dbpassword>@localhost --deploy-as=root:<password> -e <encryption_type> -m <management_server_key> -k <database_key> -r <region_id>

  3. Start mgmt server

  4. Using addRegion API, add region 1 to region 2 and also region 2 to region 1.

  5. copy account/user/domain tables from Region1 DB to Region2 DB:

  • mysqldump -u cloud -p -h <region1_db_host> cloud account user domain > region1.sql
  • mysql -u cloud -p -h <region2_db_host> cloud < region1.sql      

  6. Remove project accounts after copying: 

  • mysql> delete from account where type = 5; 

  7. Set default zone as null 

  • mysql> update account set default_zone_id = null; 

  8. Restart mgmt servers in region 2

  Adding 3rd and subsequent Regions

  1. Install CS in all new regions

  2. While installing database set region_id using -r option in cloud-setup-databases script (Make sure _database_key _is same across all regions).

  cloud-setup-databases cloud:<dbpassword>@localhost --deploy-as=root:<password> -e <encryption_type> -m <management_server_key> -k <database_key> -r <region_id>

  3. Start mgmt server

  4. Using addRegion API, add existing regions to region n and also region n to all existing regions

  5. copy account/user/domain tables from any existing Region DB to RegionN DB. 

  • mysqldump -u cloud -p -h <region1_db_host> cloud account user domain > region1.sql
  • mysql -u cloud -p -h <regionN_db_host> cloud < region1.sql  

  6. Remove project accounts after copying:

  • mysql> delete from account where type = 5;

  7. Set default zone as null

  • mysql> update account set default_zone_id = null;

  8. Restart mgmt servers in region N

  Remove Region

  1. Remove region from all other regions using removeRegion API

  Appendix

  Appendix A:

  Appendix B: