You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 4 Next »

Highly Available Client Event Mechanism.

One of the key feature of Geode is reliable asynchronous event notifications. The publish/subscribe systems offer a data distribution service where new events are published into the system and routed to all interested subscribers in a reliable manner.

The clients can subscribe interests using keys or CQ.

The product achieves reliability and HA (high availability) of the subscription event using HARegionQueue.

HARegionQueue

It’s an implementation of a queue using a Geode region as the underlying data structure.

When a client connects to the server with subscription enabled flag, a HARegionQueue for that client-proxy is created on the server side.

 

Class Diagrams Server to Client Queues - Class Diagram CacheClientNotifier Singleton for a Cache. CacheClientProxy One for each client.Keeps track of a the subscriptions for the client. MessageDispatcher Thread that reads events from the queue. HARegionQueue Holds the list of events todispatch to a single client. HAContainerMap Holds the actual values for events to dispatch.Used to store only a single copy of each valueacross all queues. HARegion

Sequence Diagrams

Put into Queue (for Partitioned Regions) user user PR Primary PR Primary PR Secondary PR Secondary Adjunct Receiver Adjunct Receiver CacheClientNotifier CacheClientNotifier CacheClientProxy CacheClientProxy MessageDispatcher MessageDispatcher HARegionQueue HARegionQueue put UpdateMessage PutMessage The primary notifies all members that haveclients with interest notifyClients() notifyClients() All members with interestcall notifyClients locally notifyClients() deliverMessage() enqueueMessage() basicPut() 

Adding events to HARegionQueue:

When an operation is performed on the cache, the “LocalRegion.notifyBridgeClients()” is called to see if there are interested clients (“generateLocalFilterRouting()”) and to deliver the event (“CacheClientNotifier.notifyClients()”). Based on the interest satisfied by the client, the ClientMessage is created and added to the HARegionQueue of the interested client (“CacheClientProxy.enqueueMessage()”).

Once the event is added to the HARegionQueue the cache operation thread returns to the caller.

Delivering/Dispatching Events to Client:

The client events are delivered asynchronously to the clients using MessageDispatcher thread (CacheClientProxy.MessageDispatcher). The dispatcher thread peeks and event from the HARegionQueue and delivers it to the client.

Ack from Client:

Client:
Upon receiving the event, the client updates its local cache or invokes the CqListeners (“CacheClientUpdater.processMessages()”).

Also the client acknowledges the server periodically with the received event Id (ThreadId, sequence Id); for removal events from HARegionQueue.

Server:
The server process the ack command and maintains it in HARegionQueue.ackedEvents map, uses this for message removal.

Removing Events:

The event from HARegionQueue is removed based on the client acknowledgement (“HAregionQueue.remove()”).

Reliable Event Delivery:

The reliable event delivery (in case of node failure) is achieved by subscription redundancy level. Client can configure to have redundant event queues created on multiple server nodes by setting the redundancy level.

When redundancy is set; the HARegionQueues are created on multiple servers; one of the queue is treated as primary queue and others as secondary queues.

When the cache operation is performed, it is distributed to other nodes; based on the interest subscription the events are added to primary and secondary HARegionQueues (as explained above). The HARegionQueue’s don’t replicate the events between them.

The events are dispatched to clients from primary HARegionQueue, if the node hosting the primary queue goes down, one of the secondary queue becomes primary and starts dispatching the events. During this scenario it could so happen that, duplicate events are sent to client. At the client side checks are made for duplicate events and are ignored.

When the message is delivered and successfully removed from primary HARegionQueue, the acknowledgement is sent to other server hosting the queues (“QueueRemovalMessage”).

Creating Secondary HAQueues:

When the secondary queues are created; in order to be in sync with primary queue, they perform initial data replication from one of the existing queue (using backing region’s GII mechanism - “HARegionQueue.createHARegion()”).

During queue initialization, If there are any events satisfying the interest criteria; those events are queued temporarily in “CacheClientProxy.queuedEvents”, once the initialization is over, the events from the temporary queue is added to the HARegionQueue for delivery.

After the events are delivered and removed from primary queue (node), it sends the “QueueRemovalMessage” to secondary queues (nodes); as part of message processing the secondary queue removes events from its queue(“QueueRemovalMessage.process().removeDispatchedEvents()”.

Memory Optimization:

Based on number of client queues, and the rate at which events are delivered, the individual queues may consume lot of memory; in order to reduce the memory footprint the events are stored in container (“HAContainer”) and all the client queues will refer to this instead of having their own copy (“HARegionQueue.putEntryConditionallyIntoHAContainer()”).

The reason for doing this is, as we have seen, in most of the cases clients will have common interests, having a single copy of the client message will reduce the memory significantly.

Once event is delivered to all the interested client; the event from HAContainer is removed (“HARegionQueue.decAndRemoveFromHAContainer()”).

Code path for adding event to HAContainer:
HARegionQueue.basicPut()->putObject()->putEventInHARegion()->putEntryConditionallyIntoHAContainer().

Event Conflation

The clients can configure events to be conflated; in this case the old events in the HARegionQueue will be destroyed and new event is added to the tail of the Queue.

This is done at the end of “HARegionQueue.putObject()”.

 

 

 

 

 

 

  • No labels