You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 22 Next »

Bug Reference

https://issues.apache.org/jira/browse/CLOUDSTACK-4992

Branch

4.4 (question)

Introduction

Currently, under the environment of cloudstack with multiple regions, each region has its own management server running with a separate database. So if we want to support multiple regions and provide one point of entry for a customer, we need to duplicate domain/account/user information of that customer to all of the databases of regions the customer accesses, which will cause data discrepancies when users update those data independently in each management server.

Purpose

I'd like to provide a way to sync up the data using the messaging system introduced in 4.1.0. Using the events from each management server, updates from each region can be propagated to the rest regions and they can be executed accordingly.

Architecture and Design description

All management servers allow manual changes and any change in any server will be propagated to the rest of servers.

Restrictions

  1. The duplicated resources in each region have different uuids, which causes same domain/account/user resources in other regions to have uuids different from 'entityuuid's in the event messages.
  2. The old resource names are not available after 'update' tasks on resources are completed, which makes other regions unable to find the duplicated resources to update.
  3. It is not possible for the management server to know if the requested task is from UI by manual or from another region for sync-up

Overall Architecture

  • Each event subscriber find entity detail along with those of its parents, which resolves #1 restriction above.
  • When each command store its 'uuid' in the UserContext, it also stores the old entity name, which resolves #2 restriction above.
  • The subscribers that send corresponding task requests to other regions using API just ignore when the requests have been failed.
EventBus Class

A new class that extends EventBus class to register domain/account/user event subscribers. 

  • MultiRegionEventBus
    • It registers all subscribers with appropriate EventTopic

public class MultiRegionEventBus extends RabbitMQEventBus

    public boolean configure(String name, Map<String, Object> params) throws ConfigurationException       

        EventTopic topic = new EventTopic("ActionEvent", "*", "Domain", "*", "*");
        subscribe(topic, new DomainSubscriber(1));

                   ......

        EventTopic topic = new EventTopic("ActionEvent", "*", "Account", "*", "*");
        subscribe(topic, new AccountSubscriber(2));

                   .....

        EventTopic topic = new EventTopic("ActionEvent", "*", "User", "*", "*");
        subscribe(topic, new UserSubscriber(3));

 

                   ......

EventSubscriber Class

A new class that extends EventSubscriber class to process events of domain/account/user.

  • MultiRegionSubscriber
    • Base class that below 3 classes extends from.
    • It maintains Dao class objects to find details of the entity transferred by the given event.
    • When finding entity details, it uses methods whose return values include the already removed because the given entity may have been deleted when the event is 'DELETE'.

public class MultiRegionSubscriber  implements EventSubscriber

    protected DomainDao domainDao;
    protected AccountDao accountDao;
    protected UserDao userDao;

 

    protected boolean isCompleted(String status)
        return (status != null && status.equals("Completed"));

 

    protected boolean isExecutable()
        String status = this.descMap.get("status");
        if (!isCompleted(status)) return false;

        String entityUUID = this.descMap.get("entityuuid");
        if (entityUUID == null || entityUUID.equals(""))
            return false;
        return true;


  • DomainSubscriber
    • It listens only to domain events
    • Send a request to other regions to propagate the task of the current received event using the detail information of the entity
    • Send requests to management server of other regions only when the status of the given event is 'Completed' and the event has a valid 'entityuuid'

public class DomainSubscriber extends MultiRegionSubscriber

   

    @Override
    public void onEvent(Event event)
        super.onEvent(event);
        if (!isExecutable()) return;
        process(event);


    protected void process(Event event)

        String entityUUID = this.descMap.get("entityuuid");
        String oldDomainName = this.descMap.get("oldentityname");
        Domain domain = this.domainDao.findByUuidIncludingRemoved(entityUUID);
        Domain parentDomain = this.domainDao.findByIdIncludingRemoved(domain.getParent());

                    .......

        String methodName = event.getEventType().split("-")[1].toLowerCase();

         DomainService domainService = new DomainService(hostName, userName, password);
         Method method = domainService.getClass().getMethod(methodName, Domain.class, Domain.class, String.class);
         method.invoke(domainService, domain, parentDomain, oldDomainName);

 

  • AccountSubscriber
    • It listens only to account events
    • Send a request to other regions to propagate the task of the current received event using the detail information of the entity
    • Send requests to management server of other regions only when the status of the given event is 'Completed' and the event has a valid 'entityuuid'

public class AccountSubscriber extends MultiRegionSubscriber

 

    @Override
    public void onEvent(Event event)
        super.onEvent(event);
        if (!isExecutable()) return;
        process(event);


    protected void process(Event event)   

        String entityUUID = this.descMap.get("entityuuid");
        String oldAccountName = this.descMap.get("oldentityname");
        User user = null;
        Account account = this.accountDao.findByUuidIncludingRemoved(entityUUID);
        Domain domain = this.domainDao.findByIdIncludingRemoved(account.getDomainId());
        List<UserVO> users = this.userDao.listByAccount(account.getAccountId());
        if (users.size() > 0)    user = users.get(0);

                   ...............

 

        String methodName = event.getEventType().split("-")[1].toLowerCase();

        AccountService accountService = new AccountService(hostName, userName, password);
        Method method = accountService.getClass().getMethod(methodName, User.class, Account.class, Domain.class, String.class);
        method.invoke(accountService, user, account, domain, oldAccountName);

 

  • UserSubscriber
    • It listens only to user events
    • Send a request to other regions to propagate the task of the current received event using the detail information of the entity
    • Send requests to management server of other regions only when the status of the given event is 'Completed' and the event has a valid 'entityuuid'

public class UserSubscriber extends MultiRegionSubscriber

 

    @Override
    public void onEvent(Event event)
        super.onEvent(event);
        if (!isExecutable()) return;
        process(event);


    protected void process(Event event)  

        String entityUUID = this.descMap.get("entityuuid");
        String oldUserName = this.descMap.get("oldentityname");
        User user = this.userDao.findByUuidIncludingRemoved(entityUUID);
        Account account = this.accountDao.findByIdIncludingRemoved(user.getAccountId());
        Domain domain = this.domainDao.findByIdIncludingRemoved(account.getDomainId());

                   ...............

 

        String methodName = event.getEventType().split("-")[1].toLowerCase();

        UserService userService = new UserService(hostName, userName, password);
        Method method = userService.getClass().getMethod(methodName, User.class, Account.class, Domain.class, String.class);
        method.invoke(userService, user, account, domain, oldUserName);

Service Class

A new class to send requests to remote regions using cloudstack API

  • BaseService : Base class that below 3 classes extend from.
  • DomainService : Service class to call API interfaces related with domains
  • AccountService : Service class to call API interfaces related with accounts
  • UserService : Service class to call API interfaces related with users

 

Full Scan Support

Schema Changes

To resolve the conflicts among the resources, below fields are added

  1. Domain
    • created : timestamp when this resource is created
    • modified : timestamp when this resources is updated
  2. Account
    • created : timestamp when this resource is created
    • modified : timestamp when this resource is updated/disabled/locked/enabled
  3. User
    • created : already exists
    • modified : timestamp when this resource is updated/disable/enabled
Event Description Change

For events to provide details of resources that have been created/deleted, their names along with those of parents will be added in the description of event. (the portion underlined are newly added)

  • CREATE Event

Successfully completed creating Domain. Domain Name: domain1, Parent DomainId :1, Parent Domain Name:domain_parent

Successfully completed creating Account. Account Name: account1, Domain Id:8, Domain Name: domain1

Successfully completed creating User. User Name: user1, FirstName :first, LastName: last, Account Name: account1, Domain Name: domain1

  • DELETE Event

Successfully completed deleting Domain. Domain Id: 2, Domain Name: domain1, Parent Domain Name:domain_parent

Successfully completed deleting account. Account Id: 4, Account Name: account1, Domain Name: domain1

Successfully completed deleting User. UserId: 6, User Name: user1, Account Name: account1, Domain Name: domain1

Overall Logic

Process this below routines in each region

current_region = where this routine is processed

for remote_region in other_regions

    for remote_resource (domain/account/user) in remote_region.find_remote_resource_list( )

        resource = current_region.find_same_resource( )

        if resource != null && resource.status == remote_resource.status

            continue

        resource = current_region.find_the_latest_same_resource_including_removed( )

        if resource != null && resource.removed > remote_resource.created

            continue    // this difference will be resolved in the remote_region

        resource = current_region.create_same_resource( set resource.created as remote_resource.created )

        if resource.status == remote_resource.status

            continue

        if resource.modified > remote_resource.modified

            continue      // this difference will be resolved in the remote_region

        set resource.status as remote_resource.status


for resource in current_region.not_compared_resource_list

    // resource can be either of below 2 cases

    //    1. All remote regions have failed to create this resource : do nothing because this resource will be synced up in remote regions

    //    2. Only current region has failed to remove this resource : just delete this resource

    deleted = false

    for remote_region in other_regions

        remote_event_list = remote_region.find_event_list (

                                                 type = "<RESOURCE>.DELETE"

                                                 keyword = "Successfully completed deleting <resource>"

                                                 startdate = resource.created)

        if the event of resource is in the event list

                deleted = true

    if deleted == true

        current_region.delete (resource)

References

Github Repository

https://github.com/alexoughsg/Albatross/tree/albatross

First Implementation 

https://github.com/alexoughsg/Albatross/pull/1

Document History

Glossary

Use cases

Appendix

 

  • No labels