Page History

...

https://issues.apache.org/jira/browse/CLOUDSTACK-4992

Branch

4.3 4

Introduction

Currently, under the environment of cloudstack with multiple regions, each region has its own management server running with a separate database, which will cause data discrepancies when users create/update/delete domain/account/user data independently in each management server. So if we want to support multiple regions and provide one point of entry for a each customer, we need to duplicate domain/account/user information of that each customer to all of the databases of regions the customer accesses, which will cause data discrepancies when users update those data independently in each management server.

Purpose

I'd like to provide a way to sync up the data using the messaging system introduced in 4.1.0. Using the events from each management server, updates from each region can be propagated to the rest regions and they can be executed accordingly.

Architecture and Design description

There can be 2 different approaches.

...

All management servers allow manual changes and any change in any server will be propagated to the rest of servers.

Restrictions

The duplicate in each region has a different uuid, which causes same domain/account/user names to have different 'entityuuid's in the event messages. Under this restriction, in the events of 'XXXXX-DELETE', it is not possible to find target resource information like name because the resource already has been deleted from its region.
In #2 approach, the messaging system does not provide a way to distinguish events from either manual jobs or triggered automatic processing.

Feature Specifications

For #1 restriction, we may add more information like actual resource and its parent names in the current event message by either way
- update action command classes like 'CreateAccountCmd', 'CreateDomainCmd', etc.
- create new extended classes from any, if not all, classes of 'ActionEventInterceptor', 'RabbitMQEventBus', etc.
For #2 restriction, we may provide a logging system to store details of completed jobs, which can give a decent way to trace events.

Overall Architecture

We may provide a queue where the 'event handler' just stores the messages whenever it receives and the 'message processor' processes the stored messages sequentially.
This way, both the event handler is not blocked and the messages are guaranteed to be processed in the same order compared to them being processed in threads, and the failed messages can be retried.

Master - Slave Architecture

Tables

region : stores information of all regions especially authentication information to call API

Field	Type	Desc
name	char(255)	name of the region
host	char(255)	API server
root_url	char(255)	root url for API Interfaces
user_name	char(255)	user name for API call authentication
password	char(255)	password for API call authentication
is_master	boolean	true if this region is the master

event_log : a queue where the received messages are stored

Field	Type	Desc
routing_key	char(255)	routing key of an event message
body	char(255)	body of an event message
created_time	datetime	when this record is created
processed_time	datetime	when this record was processed
result	boolean	true if succeeded
message	char(255)	error message

Event Handler
- This is a subscriber to the message queue of the EventNotificationBus.
- It filters only messages of 'Completed ActionEvent' of 'domain', 'account' and 'user' objects
- Once it gets a message
  - stores 'routing_key' and 'body' of that message in the 'event_log' table
  - launches 'Message Processor' if it is not currently working
  - waits for next message
Message Processor
- This is a processor to sync up the completed action events of the master.
- Once launched
  - goes through the 'event_log' table to see if there are new and/or failed tasks
  - for each new and/or failed task,
    - calls the corresponding API interface to each slave with the task information
    - in case of 'DELETE' task, sync up overall data of each slave to those of the master because of #1 restriction
  - goes through the 'event_log' table again until there are no new or failed tasks

Multiple Source Architecture

Tables
- Working in progress ...
Event Handler
- Working in progress ...
Message Processor
- Working in progress ...

References

resources have different uuids in different regions even if they are identical.
It is not possible to determine the time orders of operations in the same resource but in different regions
The name of resources can be changed, but the old names are not maintained.

Solutions

Introduced a resource map table to maintain the resource uuids for all regions.
1. 'source' : uuid of a resource in this local region
2. 'region_id' : id of a remote region in 'region' table
3. 'uuid' : uuid of the same resource in the remote region whose id is 'region_id'
4. 'created' : timestamp when this map is created
5. 'modified' : timestamp when this map is last modified
6. 'removed' : timestamp when this map is removed
Added 'created', 'modified' fields in domain/account/user objects to track the time when operations occur.
Added 'Old Entity Name' and 'New Entity Name' in the event description of update events.

Real Time Synchronization

MultiRegionEventBus : A new event bus to register subscribers that catch action events of domain/account/user objects.
Domain/Account/User Subscribers that subscribe action events to send the changes in the local resources to remote regions using cloudstack API.
The subscribers that send corresponding task requests to other regions using API just ignore when the requests have been failed.

EventBus Class

A new class that extends EventBus class to register domain/account/user action event subscribers.

Server/resources/META-INF/cloudstack/core/spring-event-bus-context.xml

<bean id="eventNotificationBus" class="org.apache.cloudstack.mom.rabbitmq.MultiRegionEventBus">
<property name="name" value="eventNotificationBus" />
<property name="server" value="localhost" />
<property name="port" value="5672" />
<property name="username" value="guest" />
<property name="password" value="guest" />
<property name="exchange" value="cloudstack-events" />
</bean>

MultiRegionEventBus

public class MultiRegionEventBus extends RabbitMQEventBus
public boolean configure(String name, Map<String, Object> params) throws ConfigurationException
EventTopic topic = new EventTopic("ActionEvent", "*", "Domain", "*", "*");
  subscribe(topic, new DomainSubscriber(1));
......
  EventTopic topic = new EventTopic("ActionEvent", "*", "Account", "*", "*");
  subscribe(topic, new AccountSubscriber(2));
.....
  EventTopic topic = new EventTopic("ActionEvent", "*", "User", "*", "*");
  subscribe(topic, new UserSubscriber(3));

......

EventSubscriber Classes

New classes that catch action events of domain/account/user and request the remote regions to apply the same changes.

MultiRegionSubscriber
- Base class that below 3 classes are extended from.
- It defined methods to determine if the events are valid

public class MultiRegionSubscriber implements EventSubscriber
protected DomainDao domainDao;
protected AccountDao accountDao;
protected UserDao userDao;

protected boolean isCompleted(String status)
      return (status != null && status.equals("Completed"));

  protected boolean isExecutable()
  String status = this.descMap.get("status");
  if (!isCompleted(status)) return false;
  String entityUUID = this.descMap.get("entityuuid");
  if (entityUUID == null || entityUUID.equals(""))
  return false;
  return true;

DomainSubscriber
- It listens only to domain action events.
- Send a request to remote regions to propagate the local domain changes of the received events if they are valid.

public class DomainSubscriber extends MultiRegionSubscriber

  @Override
  public void onEvent(Event event)
      super.onEvent(event);
      if (!isExecutable()) return;
      process(event);

  protected void process(Event event)
String entityUUID = this.descMap.get("entityuuid");
  String oldDomainName = this.descMap.get("oldentityname");
  Domain domain = this.domainDao.findByUuidIncludingRemoved(entityUUID);
  Domain parentDomain = this.domainDao.findByIdIncludingRemoved(domain.getParent());
.......
  String methodName = event.getEventType().split("-")[1].toLowerCase();
DomainService domainService = new DomainService(hostName, userName, password);
Method method = domainService.getClass().getMethod(methodName, Domain.class, Domain.class, String.class);
method.invoke(domainService, domain, parentDomain, oldDomainName);

AccountSubscriber
- It listens only to account action events.
- Send a request to remote regions to propagate the local account changes of the received events if they are valid.

public class AccountSubscriber extends MultiRegionSubscriber

  @Override
  public void onEvent(Event event)
      super.onEvent(event);
      if (!isExecutable()) return;
      process(event);

  protected void process(Event event)
String entityUUID = this.descMap.get("entityuuid");
  String oldAccountName = this.descMap.get("oldentityname");
  User user = null;
  Account account = this.accountDao.findByUuidIncludingRemoved(entityUUID);
  Domain domain = this.domainDao.findByIdIncludingRemoved(account.getDomainId());
  List<UserVO> users = this.userDao.listByAccount(account.getAccountId());
  if (users.size() > 0) user = users.get(0);
...............

  String methodName = event.getEventType().split("-")[1].toLowerCase();
AccountService accountService = new AccountService(hostName, userName, password);
  Method method = accountService.getClass().getMethod(methodName, User.class, Account.class, Domain.class, String.class);
  method.invoke(accountService, user, account, domain, oldAccountName);

UserSubscriber
- It listens only to user action events
- Send a request to remote regions to propagate the local user changes of the received events if they are valid.

public class UserSubscriber extends MultiRegionSubscriber

  @Override
  public void onEvent(Event event)
      super.onEvent(event);
      if (!isExecutable()) return;
      process(event);

  protected void process(Event event)
String entityUUID = this.descMap.get("entityuuid");
  String oldUserName = this.descMap.get("oldentityname");
  User user = this.userDao.findByUuidIncludingRemoved(entityUUID);
  Account account = this.accountDao.findByIdIncludingRemoved(user.getAccountId());
  Domain domain = this.domainDao.findByIdIncludingRemoved(account.getDomainId());
...............

  String methodName = event.getEventType().split("-")[1].toLowerCase();
  UserService userService = new UserService(hostName, userName, password);
  Method method = userService.getClass().getMethod(methodName, User.class, Account.class, Domain.class, String.class);
  method.invoke(userService, user, account, domain, oldUserName);

Service Classes

New classes to send requests to remote regions using cloudstack API

BaseService : Base class that below 3 classes are extended from.
DomainService : Service class to call API interfaces related with domains
AccountService : Service class to call API interfaces related with accounts
UserService : Service class to call API interfaces related with users

Full Scan

Global Variable

'region.full.scan.interval' : time interval in milliseconds to run the full scan

Overall Logic

get the list of local resources
for each remote region
get the list of remote resources
for each resource in the local list
if there is a remote resource that is linked to this local resource
compare the 'modified' datetimes of local & remote resources
if they are same, no work
if both attribute values are same, no work
otherwise,
if the remote is after the local, UPDATE local attribute values to those of the remote including 'modified' timestamp
if the local is after, no work (This will be sync'ed up in the remote region)
remove this local from the local list
remove this remote from the remote list
otherwise, no work
for local resources that are still in the local list
find the latest removal event of the linked resource in the remote region
if an event is found in the remote region and the 'created' timestamp of the removal event is later than the 'created' timestamp of this local resource,
remove the local resource with remote event 'created' timestamp
remove this local from the local list
otherwise, no work
for remote resources that are still in the remote list
find if the local resource was removed
if it was removed and the 'removed' timestamp is later than the 'created' of the remote,
don't create this in the local region
otherwise,
create this resource in the local region with remote resource attribute values including 'created', 'modified'
remove this remote from the remote list

Database Schema Changes

domain

Newly added fields
- created
- modified

account

Newly added fields
- created
- modified

user

Newly added field
- modified

region

Newly added fields
- userName
- password
- active

rmap

Newly created with these fields
- source
- region_id
- uuid
- created
- modified
- removed

Event Description Change

To provide details of resources that have been created/modified/deleted, the resource names along with those of parents are added in the description of event. (the portion underlined are newly added)

CREATE Event

Successfully completed creating Domain. Domain Name: domain1, Parent DomainId :1, Domain Path:domain_path
Successfully completed creating Account. Account Name: account1, Domain Id:8, Domain Path:domain_path
Successfully completed creating User. User Name: user1, FirstName :first, LastName: last, Account Name: account1, Domain Path:domain_path

UPDATE Event

Successfully completed updating Domain. Domain Id: 25, Domain Path:domain_path, Old Entity Name:old_name, New Entity Name:new_name
Successfully completed updating account, Account Name:account1, Domain Path:domain_path, Old Entity Name:old_name, New Entity Name:new_name
Successfully completed updating User. UserId: 34, User Name:user1, Account Name:account, Domain Path:domain_path, Old Entity Name:old_name, New Entity Name:new_name

DELETE Event

Successfully completed deleting Domain. Domain Id: 2, Domain Path:domain_path
Successfully completed deleting account. Account Id: 4, Account Name: account1, Domain Path:domain_path
Successfully completed deleting User. UserId: 6, User Name: user1, Account Name: account1, Domain Path:domain_path

API Changes

listDomains

Newly added attributes
- created : the timestamp when this domain was created
- modified : the timestamp when this domain was last modified

listAccounts

Newly added attributes
- path : the path of the domain where this account belong
- created : the timestamp when this account was created
- modified : the timestamp when this account was last modified

list Users

Newly added attributes
- path : the path of the domain where this user's account belong
- modified : the timestamp when this user was last modified

Space shortcuts

Child pages

Page History

Versions Compared

Old Version 5

New Version Current

Key

Branch

Introduction

Purpose

Architecture and Design description

Restrictions

Feature Specifications

Overall Architecture

Master - Slave Architecture

Multiple Source Architecture

References

Solutions

Real Time Synchronization

EventBus Class

EventSubscriber Classes

Service Classes

Full Scan

Global Variable

Overall Logic

Database Schema Changes

domain

account

user

region

rmap

Event Description Change

API Changes

listDomains

listAccounts

list Users

References

Github Repository

https://github.com/alexoughsg/Albatross/tree/albatross

Pull Request

Document History

Glossary

Use cases

Appendix