Domain-Account-User Sync Up Among Multiple Regions

Bug Reference

https://issues.apache.org/jira/browse/CLOUDSTACK-4992

Branch

4.4

Introduction

Currently, under the environment of cloudstack with multiple regions, each region has its own management server running with a separate database, which will cause data discrepancies when users create/update/delete domain/account/user data independently in each management server. So if we want to support multiple regions and provide one point of entry for each customer, we need to duplicate domain/account/user information of each customer to all of the regions the customer accesses.

Purpose

I'd like to provide a way to sync up the data using the messaging system introduced in 4.1.0. Using the events from each management server, updates from each region can be propagated to the rest regions and they can be executed accordingly.

Architecture and Design description

All management servers allow manual changes and any change in any server will be propagated to the rest of servers.

Restrictions

The resources have different uuids in different regions even if they are identical.
It is not possible to determine the time orders of operations in the same resource but in different regions
The name of resources can be changed, but the old names are not maintained.

Solutions

Introduced a resource map table to maintain the resource uuids for all regions.
1. 'source' : uuid of a resource in this local region
2. 'region_id' : id of a remote region in 'region' table
3. 'uuid' : uuid of the same resource in the remote region whose id is 'region_id'
4. 'created' : timestamp when this map is created
5. 'modified' : timestamp when this map is last modified
6. 'removed' : timestamp when this map is removed
Added 'created', 'modified' fields in domain/account/user objects to track the time when operations occur.
Added 'Old Entity Name' and 'New Entity Name' in the event description of update events.

Real Time Synchronization

MultiRegionEventBus : A new event bus to register subscribers that catch action events of domain/account/user objects.
Domain/Account/User Subscribers that subscribe action events to send the changes in the local resources to remote regions using cloudstack API.
The subscribers that send corresponding task requests to other regions using API just ignore when the requests have been failed.

EventBus Class

A new class that extends EventBus class to register domain/account/user action event subscribers.

Server/resources/META-INF/cloudstack/core/spring-event-bus-context.xml

<bean id="eventNotificationBus" class="org.apache.cloudstack.mom.rabbitmq.MultiRegionEventBus">
<property name="name" value="eventNotificationBus" />
<property name="server" value="localhost" />
<property name="port" value="5672" />
<property name="username" value="guest" />
<property name="password" value="guest" />
<property name="exchange" value="cloudstack-events" />
</bean>

MultiRegionEventBus

public class MultiRegionEventBus extends RabbitMQEventBus
public boolean configure(String name, Map<String, Object> params) throws ConfigurationException
EventTopic topic = new EventTopic("ActionEvent", "*", "Domain", "*", "*");
  subscribe(topic, new DomainSubscriber(1));
......
  EventTopic topic = new EventTopic("ActionEvent", "*", "Account", "*", "*");
  subscribe(topic, new AccountSubscriber(2));
.....
  EventTopic topic = new EventTopic("ActionEvent", "*", "User", "*", "*");
  subscribe(topic, new UserSubscriber(3));

......

EventSubscriber Classes

New classes that catch action events of domain/account/user and request the remote regions to apply the same changes.

MultiRegionSubscriber
- Base class that below 3 classes are extended from.
- It defined methods to determine if the events are valid

public class MultiRegionSubscriber implements EventSubscriber
protected DomainDao domainDao;
protected AccountDao accountDao;
protected UserDao userDao;

protected boolean isCompleted(String status)
      return (status != null && status.equals("Completed"));

  protected boolean isExecutable()
  String status = this.descMap.get("status");
  if (!isCompleted(status)) return false;
  String entityUUID = this.descMap.get("entityuuid");
  if (entityUUID == null || entityUUID.equals(""))
  return false;
  return true;

DomainSubscriber
- It listens only to domain action events.
- Send a request to remote regions to propagate the local domain changes of the received events if they are valid.

public class DomainSubscriber extends MultiRegionSubscriber

  @Override
  public void onEvent(Event event)
      super.onEvent(event);
      if (!isExecutable()) return;
      process(event);

  protected void process(Event event)
String entityUUID = this.descMap.get("entityuuid");
  String oldDomainName = this.descMap.get("oldentityname");
  Domain domain = this.domainDao.findByUuidIncludingRemoved(entityUUID);
  Domain parentDomain = this.domainDao.findByIdIncludingRemoved(domain.getParent());
.......
  String methodName = event.getEventType().split("-")[1].toLowerCase();
DomainService domainService = new DomainService(hostName, userName, password);
Method method = domainService.getClass().getMethod(methodName, Domain.class, Domain.class, String.class);
method.invoke(domainService, domain, parentDomain, oldDomainName);

AccountSubscriber
- It listens only to account action events.
- Send a request to remote regions to propagate the local account changes of the received events if they are valid.

public class AccountSubscriber extends MultiRegionSubscriber

  @Override
  public void onEvent(Event event)
      super.onEvent(event);
      if (!isExecutable()) return;
      process(event);

  protected void process(Event event)
String entityUUID = this.descMap.get("entityuuid");
  String oldAccountName = this.descMap.get("oldentityname");
  User user = null;
  Account account = this.accountDao.findByUuidIncludingRemoved(entityUUID);
  Domain domain = this.domainDao.findByIdIncludingRemoved(account.getDomainId());
  List<UserVO> users = this.userDao.listByAccount(account.getAccountId());
  if (users.size() > 0) user = users.get(0);
...............

  String methodName = event.getEventType().split("-")[1].toLowerCase();
AccountService accountService = new AccountService(hostName, userName, password);
  Method method = accountService.getClass().getMethod(methodName, User.class, Account.class, Domain.class, String.class);
  method.invoke(accountService, user, account, domain, oldAccountName);

UserSubscriber
- It listens only to user action events
- Send a request to remote regions to propagate the local user changes of the received events if they are valid.

public class UserSubscriber extends MultiRegionSubscriber

  @Override
  public void onEvent(Event event)
      super.onEvent(event);
      if (!isExecutable()) return;
      process(event);

  protected void process(Event event)
String entityUUID = this.descMap.get("entityuuid");
  String oldUserName = this.descMap.get("oldentityname");
  User user = this.userDao.findByUuidIncludingRemoved(entityUUID);
  Account account = this.accountDao.findByIdIncludingRemoved(user.getAccountId());
  Domain domain = this.domainDao.findByIdIncludingRemoved(account.getDomainId());
...............

  String methodName = event.getEventType().split("-")[1].toLowerCase();
  UserService userService = new UserService(hostName, userName, password);
  Method method = userService.getClass().getMethod(methodName, User.class, Account.class, Domain.class, String.class);
  method.invoke(userService, user, account, domain, oldUserName);

Service Classes

New classes to send requests to remote regions using cloudstack API

BaseService : Base class that below 3 classes are extended from.
DomainService : Service class to call API interfaces related with domains
AccountService : Service class to call API interfaces related with accounts
UserService : Service class to call API interfaces related with users

Full Scan

Global Variable

'region.full.scan.interval' : time interval in milliseconds to run the full scan

Overall Logic

get the list of local resources
for each remote region
get the list of remote resources
for each resource in the local list
if there is a remote resource that is linked to this local resource
compare the 'modified' datetimes of local & remote resources
if they are same, no work
if both attribute values are same, no work
otherwise,
if the remote is after the local, UPDATE local attribute values to those of the remote including 'modified' timestamp
if the local is after, no work (This will be sync'ed up in the remote region)
remove this local from the local list
remove this remote from the remote list
otherwise, no work
for local resources that are still in the local list
find the latest removal event of the linked resource in the remote region
if an event is found in the remote region and the 'created' timestamp of the removal event is later than the 'created' timestamp of this local resource,
remove the local resource with remote event 'created' timestamp
remove this local from the local list
otherwise, no work
for remote resources that are still in the remote list
find if the local resource was removed
if it was removed and the 'removed' timestamp is later than the 'created' of the remote,
don't create this in the local region
otherwise,
create this resource in the local region with remote resource attribute values including 'created', 'modified'
remove this remote from the remote list

Database Schema Changes

domain

Newly added fields
- created
- modified

account

Newly added fields
- created
- modified

user

Newly added field
- modified

region

Newly added fields
- userName
- password
- active

rmap

Newly created with these fields
- source
- region_id
- uuid
- created
- modified
- removed

Event Description Change

To provide details of resources that have been created/modified/deleted, the resource names along with those of parents are added in the description of event. (the portion underlined are newly added)

CREATE Event

Successfully completed creating Domain. Domain Name: domain1, Parent DomainId :1, Domain Path:domain_path
Successfully completed creating Account. Account Name: account1, Domain Id:8, Domain Path:domain_path
Successfully completed creating User. User Name: user1, FirstName :first, LastName: last, Account Name: account1, Domain Path:domain_path

UPDATE Event

Successfully completed updating Domain. Domain Id: 25, Domain Path:domain_path, Old Entity Name:old_name, New Entity Name:new_name
Successfully completed updating account, Account Name:account1, Domain Path:domain_path, Old Entity Name:old_name, New Entity Name:new_name
Successfully completed updating User. UserId: 34, User Name:user1, Account Name:account, Domain Path:domain_path, Old Entity Name:old_name, New Entity Name:new_name

DELETE Event

Successfully completed deleting Domain. Domain Id: 2, Domain Path:domain_path
Successfully completed deleting account. Account Id: 4, Account Name: account1, Domain Path:domain_path
Successfully completed deleting User. UserId: 6, User Name: user1, Account Name: account1, Domain Path:domain_path

API Changes

listDomains

Newly added attributes
- created : the timestamp when this domain was created
- modified : the timestamp when this domain was last modified

listAccounts

Newly added attributes
- path : the path of the domain where this account belong
- created : the timestamp when this account was created
- modified : the timestamp when this account was last modified

list Users

Newly added attributes
- path : the path of the domain where this user's account belong
- modified : the timestamp when this user was last modified

Space shortcuts

Child pages