Domain-Account-User Sync Up Among Multiple Regions

Bug Reference

https://issues.apache.org/jira/browse/CLOUDSTACK-4992

Branch

4.4

Introduction

Currently, under the environment of cloudstack with multiple regions, each region has its own management server running with a separate database. So if we want to support multiple regions and provide one point of entry for a customer, we need to duplicate domain/account/user information of that customer to all of the databases of regions the customer accesses, which will cause data discrepancies when users update those data independently in each management server.

Purpose

I'd like to provide a way to sync up the data using the messaging system introduced in 4.1.0. Using the events from each management server, updates from each region can be propagated to the rest regions and they can be executed accordingly.

Architecture and Design description

All management servers allow manual changes and any change in any server will be propagated to the rest of servers.

Restrictions

The duplicated resources in each region have different uuids, which causes same domain/account/user resources in other regions to have uuids different from 'entityuuid's in the event messages.
The old resource names are not available after 'update' tasks on resources are completed, which makes other regions unable to find the duplicated resources to update.
It is not possible for the management server to know if the requested task is from UI by manual or from another region for sync-up

Overall Architecture

Each event subscriber find entity detail along with those of its parents, which resolves #1 restriction above.
When each command store its 'uuid' in the UserContext, it also stores the old entity name, which resolves #2 restriction above.
The subscribers that send corresponding task requests to other regions using API just ignore when the requests have been failed.

EventBus Class

A new class that extends EventBus class to register domain/account/user event subscribers.

MultiRegionEventBus
- It registers all subscribers with appropriate EventTopic

public class MultiRegionEventBus extends RabbitMQEventBus
public boolean configure(String name, Map<String, Object> params) throws ConfigurationException
EventTopic topic = new EventTopic("ActionEvent", "*", "Domain", "*", "*");
  subscribe(topic, new DomainSubscriber(1));
......
  EventTopic topic = new EventTopic("ActionEvent", "*", "Account", "*", "*");
  subscribe(topic, new AccountSubscriber(2));
.....
  EventTopic topic = new EventTopic("ActionEvent", "*", "User", "*", "*");
  subscribe(topic, new UserSubscriber(3));

......

EventSubscriber Class

A new class that extends EventSubscriber class to process events of domain/account/user.

MultiRegionSubscriber
- Base class that below 3 classes extends from.
- It maintains Dao class objects to find details of the entity transferred by the given event.
- When finding entity details, it uses methods whose return values include the already removed because the given entity may have been deleted when the event is 'DELETE'.

public class MultiRegionSubscriber implements EventSubscriber
protected DomainDao domainDao;
protected AccountDao accountDao;
protected UserDao userDao;

protected boolean isCompleted(String status)
      return (status != null && status.equals("Completed"));

  protected boolean isExecutable()
  String status = this.descMap.get("status");
  if (!isCompleted(status)) return false;
  String entityUUID = this.descMap.get("entityuuid");
  if (entityUUID == null || entityUUID.equals(""))
  return false;
  return true;

DomainSubscriber
- It listens only to domain events
- Send a request to other regions to propagate the task of the current received event using the detail information of the entity
- Send requests to management server of other regions only when the status of the given event is 'Completed' and the event has a valid 'entityuuid'

public class DomainSubscriber extends MultiRegionSubscriber

  @Override
  public void onEvent(Event event)
      super.onEvent(event);
      if (!isExecutable()) return;
      process(event);

  protected void process(Event event)
String entityUUID = this.descMap.get("entityuuid");
  String oldDomainName = this.descMap.get("oldentityname");
  Domain domain = this.domainDao.findByUuidIncludingRemoved(entityUUID);
  Domain parentDomain = this.domainDao.findByIdIncludingRemoved(domain.getParent());
.......
  String methodName = event.getEventType().split("-")[1].toLowerCase();
DomainService domainService = new DomainService(hostName, userName, password);
Method method = domainService.getClass().getMethod(methodName, Domain.class, Domain.class, String.class);
method.invoke(domainService, domain, parentDomain, oldDomainName);

AccountSubscriber
- It listens only to account events
- Send a request to other regions to propagate the task of the current received event using the detail information of the entity
- Send requests to management server of other regions only when the status of the given event is 'Completed' and the event has a valid 'entityuuid'

public class AccountSubscriber extends MultiRegionSubscriber

  @Override
  public void onEvent(Event event)
      super.onEvent(event);
      if (!isExecutable()) return;
      process(event);

  protected void process(Event event)
String entityUUID = this.descMap.get("entityuuid");
  String oldAccountName = this.descMap.get("oldentityname");
  User user = null;
  Account account = this.accountDao.findByUuidIncludingRemoved(entityUUID);
  Domain domain = this.domainDao.findByIdIncludingRemoved(account.getDomainId());
  List<UserVO> users = this.userDao.listByAccount(account.getAccountId());
  if (users.size() > 0) user = users.get(0);
...............

  String methodName = event.getEventType().split("-")[1].toLowerCase();
AccountService accountService = new AccountService(hostName, userName, password);
  Method method = accountService.getClass().getMethod(methodName, User.class, Account.class, Domain.class, String.class);
  method.invoke(accountService, user, account, domain, oldAccountName);

UserSubscriber
- It listens only to user events
- Send a request to other regions to propagate the task of the current received event using the detail information of the entity
- Send requests to management server of other regions only when the status of the given event is 'Completed' and the event has a valid 'entityuuid'

public class UserSubscriber extends MultiRegionSubscriber

  @Override
  public void onEvent(Event event)
      super.onEvent(event);
      if (!isExecutable()) return;
      process(event);

  protected void process(Event event)
String entityUUID = this.descMap.get("entityuuid");
  String oldUserName = this.descMap.get("oldentityname");
  User user = this.userDao.findByUuidIncludingRemoved(entityUUID);
  Account account = this.accountDao.findByIdIncludingRemoved(user.getAccountId());
  Domain domain = this.domainDao.findByIdIncludingRemoved(account.getDomainId());
...............

  String methodName = event.getEventType().split("-")[1].toLowerCase();
  UserService userService = new UserService(hostName, userName, password);
  Method method = userService.getClass().getMethod(methodName, User.class, Account.class, Domain.class, String.class);
  method.invoke(userService, user, account, domain, oldUserName);

Service Class

A new class to send requests to remote regions using cloudstack API

BaseService : Base class that below 3 classes extend from.
DomainService : Service class to call API interfaces related with domains
AccountService : Service class to call API interfaces related with accounts
UserService : Service class to call API interfaces related with users

Full Scan Support

Schema Changes

To resolve the conflicts among the resources, below fields are added

Domain
- created : timestamp when this resource is created
- modified : timestamp when this resources is updated
Account
- created : timestamp when this resource is created
- modified : timestamp when this resource is updated/disabled/locked/enabled
User
- created : already exists
- modified : timestamp when this resource is updated/disable/enabled

Event Description Change

For events to provide details of resources that have been created/deleted, their names along with those of parents will be added in the description of event. (the portion underlined are newly added)

CREATE Event

Successfully completed creating Domain. Domain Name: domain1, Parent DomainId :1, Domain Path:domain_path
Successfully completed creating Account. Account Name: account1, Domain Id:8, Domain Path:domain_path
Successfully completed creating User. User Name: user1, FirstName :first, LastName: last, Account Name: account1, Domain Path:domain_path

DELETE Event

Successfully completed deleting Domain. Domain Id: 2, Domain Path:domain_path
Successfully completed deleting account. Account Id: 4, Account Name: account1, Domain Path:domain_path
Successfully completed deleting User. UserId: 6, User Name: user1, Account Name: account1, Domain Path:domain_path

Overall Logic

Process this below routines in each region

current_region = where this routine is processed
for remote_region in other_regions
for remote_resource (domain/account/user) in remote_region.find_remote_resource_list( )
resource = current_region.find_same_resource( )
if resource != null && resource.status == remote_resource.status
continue
resource = current_region.find_the_latest_same_resource_including_removed( )
if resource != null && resource.removed > remote_resource.created
continue // this difference will be resolved in the remote_region
resource = current_region.create_same_resource( set resource.created as remote_resource.created )
if resource.status == remote_resource.status
continue
if resource.modified > remote_resource.modified
continue // this difference will be resolved in the remote_region
set resource.status as remote_resource.status

for resource in current_region.not_compared_resource_list
// resource can be either of below 2 cases
// 1. All remote regions have failed to create this resource : do nothing because this resource will be synced up in remote regions
// 2. Only current region has failed to remove this resource : just delete this resource
deleted = false
for remote_region in other_regions
remote_event_list = remote_region.find_event_list (
type = "<RESOURCE>.DELETE"
keyword = "Successfully completed deleting <resource>"
startdate = resource.created)
if the event of resource is in the event list
deleted = true
if deleted == true
current_region.delete (resource)

API Changes

listDomains

New attributes & their values
- created : the timestamp when this domain was created
- modified : the timestamp when this domain was last modified

listAccounts

New attributes & their values
- path : the path of the domain where this account belong
- created : the timestamp when this account was created
- modified : the timestamp when this account was last modified

list Users

New attribute & its value
- path : the path of the domain where this account belong
- modified : the timestamp when this user was last modified

listEvents

Currently, 'startdate' and 'enddate' parameter values are converted to the same timezone of the management server, which can cause incorrect return when the sender is in a different timezone.
To resolve this, a new Date format string ("yyyy-MM-dd HH:mm:ss z") is added for 'startdate' and 'enddate' parameter values.
- for example, "2013-01-01 12:12:12 GMT"

Space shortcuts

Child pages