EventBus

Problem

There are multiple parties like event-driven workflows systems, monitoring system, usage & billing system would be interested in getting notification of a events in the cloud in near instantaneous manner. In CloudStack event could be state change of virtual or psychical resources, an action performed by user (action events), policy based events (alerts). Currently CloudStack has no notification mechanism to let the interested parties know of the occurrence of an event. Polling by invoking listEvents and listAlerts is the only way to know occurrence of events. Also there are no externsion hooks one can implement to get the event notification. Though there is StateListener interface which has both pre and post state transition event call back methods, only user VM resource has state listener implemented. Most of the resources volume, network, nic, etc has states but does not have state machine and StateListener. So for a component interested getting notification need to get tightly coupled into orchestration workflow. For e.g, code to generate usage date (usage events) is spread across the orchestration components.

Proposal

This proposal is to introduce EventBus abstraction with in the management server and by which achieve event notifications. Scope of the event bus is only with in the management server, so only CloudStack components and extension plug-in's can be subscribers to the events. Goal of this proposal is to have bare minimum mechanism in the CloudStack code, using which more powerful notification mechanism (like SMS, email etc) can be built as CloudStack extensions.

Ideal requirements for such event bus is to be

• light weight (a library)
• provide simple interface
• decouple the publisher and subscriber so that they need not be aware of other
• powerful enough to provide filters for events of a type to wildcard (all events)
• deliver events asynchronously. Publisher publishes the event and forgets.

EventBus

Notion of EventBus shall be introduced which shall have below interface.

public interface EventBus {
    /**
     * publish an event on to the event bus
     *
     * @param event event that needs to be published on the event bus
     */
    void publish(Event event) throws EventBusException;
    /**
     * subscribe to events that matches specified event topics
     *
     * @param topic defines category and type of the events being subscribed to
     * @param subscriber subscriber that intends to receive event notification
     * @return UUID returns the subscription ID
     */
    UUID subscribe(EventTopic topic, EventSubscriber subscriber) throws EventBusException;
    /**
     * unsubscribe to events of a category and a type
     *
     * @param subscriber subscriber that intends to unsubscribe from the event notification
     */
    void unsubscribe(UUID subscriberId, EventSubscriber subscriber) throws EventBusException;
}

Following CloudStack components that generate events, shall inject EventBus and use publish method to publish an event on to event bus.

• ActionEventUtils
• UsageEventUtils
• AlertGenerator
• UserVmStateListener
• NetworkStateListener
• VolumeStateListener
• SnapshotStateListener

Below structure shall be used for the Event that is published on to event bus.

public class Event {
    String eventCategory; // describes the category of event which can be 'ActionEvent', 'AlertEvent', 'UsageEvent', 'ResourceStateEvent', 'AsyncJobEvent'
    String eventType;     // describes the type of the event with the event category. For e.g for an events of category 'ActionEvent' type could be VM.START, VM.STOP etc
    String eventSource;   // source of the event, what component of management server generated the events, plug-in which generated event etc
    String resourceType;  // describes the type of the resource for which event generated. E.g.) VirtualMachine, Network, Volume, SnapShot etc.
    String resourceUUID;  // described the UUID of the resource for which event is generated
    String description;   // general description of the event
}

EventBus shall expose subscribe interface method to which a subscriber (which needs to be a component running in management server) can register for call backs when an event occurs.  Intended subscriber shall implement EventSubscriber interface as described below.

public interface EventSubscriber {
    /**
     * Callback method. EventBus calls this method on occurrence of subscribed event
     *
     * @param event details of the event
     */
    void onEvent(Event event);
}

Subscribe method of EventBus interface shall take two parameters as described below one to provide the subscribe which will receive the call back and other parameter to define topics to which subscriber wants to subscribe to.

subscribe(EventTopic topic, EventSubscriber subscriber) throws EventBusException;

EventTopic shall have structure as described below.

public class EventTopic {
    String eventCategory; // describes the category of event subscriber is interested in, which can be 'ActionEvent', 'AlertEvent', 'UsageEvent', 'ResourceStateEvent', 'AsyncJobEvent'
    String eventType;     // describes the type of the event with the event category subscriber is interested in. For e.g for an events of category 'ActionEvent' type could be VM.START, VM.STOP etc
    String resourceType;  // describes the type of the resource for which event generated. E.g.) VirtualMachine, Network, Volume, SnapShot etc.
    String resourceUUID;  // described the UUID of the resource for which event is generated
    String eventSource;  // source of the event, what component of management server generated the events, plug-in which generated event etc
}

AMQP based event bus implementation

An AMQP plug-in will be introduced in CloudStack which will implement 'EventBus' interface, that implement semantics to EventBus interface, i.e implement publish, subscribe and unsubscribe function.

publish:

When a event is published on to event bug, with EventBus.Publish, this plug-in shall be push events generated by CloudStack to an exchange in AMQP server. Published event to the exchange in AMQP server shall have below routing key format;

  event-source.event-category.event-type.resource-type.resource-uuid

For e.g for action event generated by management server for VM start operation for VM with UUID '172dd6f8-7d36-44ef-9a03-3b342034aaa4' shall publish event to AMQP exchange with routing key;

management-server.ActionEvent.VM-START.VirtualMachine.172dd6f8-7d36-44ef-9a03-3b342034aaa4

subscribe:

when a component running along management server, subscribes to events by registering event handler with EventBus.SubScribe method, this plugin shall create a queue and bind to the exchange with binding key created from the event topics. Binding key shall of the format:

  event-source.event-category.event-type.resource-type.resource-uuid

Binding key is formed from the event topic details. if details are not provided then plugin will default to wildcard.

e.g.1) if the event topic is specified as "all action events, for the virtual machine with UUID 172dd6f8-7d36-44ef-9a03-3b342034aaa4" then binding key will formed as

          *.ActionEvent.*.VirtualMachine.172dd6f8-7d36-44ef-9a03-3b342034aaa4

Subscribing to events externally:

For the components running outside the management server context, can subscribe to events directly with the AMQP server. 

One create queue and bind to the exchange with a binding key formed as per the events interested. Binding key needs to be of the format.

  event-source.event-category.event-type.resource-type.resource-uuid

Example binding keys and the interpretation:

'*.*.*.*.*' represents a binding key that will subscribe to all the events

'*.*.*.VirtualMachine.*' represents a binding key that will subscribe to all the events related virtual machine

'*.*.*.Network.*' represents a binding key that will subscribe to all the events related virtual machine

'*.ActionEvent.*.*.*' represents a binding key that will subscribe to all the action events