Introduction

Apache Geode is a data management platform that provides real-time, consistent access to data-intensive applications throughout widely distributed cloud architectures. While it currently has high-speed client interfaces for Java, C++ and .NET there is both a need to create lighter-weight clients and a demand to access Geode from other programming languages. Unfortunately, the existing client-server protocol is undocumented.  It evolved over time and is overly complex to meet either of these needs.

This document describes a new, lighter-weight client/server protocol that may be implemented in phases, moving from a "basic client" to a more advanced "smart client".  It endeavors to provide a protocol that is also more amenable to more modern APIs such as those using asynchronous or reactive patterns.

You will notice that this document describes the protocol down to the level of byte-ordering and "bytes on the wire".  This is a description of the native serialization of the protocol that will be used for the initial implementation.  It is our intent to make this pluggable so that alternative serialization technologies may be used instead, such as Protobuf or Apache Thrift.  How that is done will be left as a goal for the architecture of the server component, with an additional goal of providing an IDL description of the protocol.

At the same time, we realize that serialization of application keys, values, callback arguments, function parameters and so forth are a separate matter and are not necessarily tied to the serialization protocol used for client/server messaging.  The initial protocol will support primitive types such as scalars, strings, and byte arrays.  It will also support JSON documents as values and convert between these and Geode PDX-serialized objects in the servers.

Goals

The high-level goals for the protocol are defined here.

Approach

 During the investigation into frameworks that would "lower the barrier of entry," it became evident that there are two approaches that can be taken to solve the problem:

1. Message Serialization Frameworks - These frameworks allow for the definition of a message in a generic IDL, the generation of language specific classes from the IDL, and the encoding/decoding of those message to be sent over a transport
2. RPC Frameworks - There frameworks provide greater coverage in the node-to-node communication: the transport layer (HTTP, TCP, UDP), the message definition in IDL with corresponding serialization mechanism and the definition of methods in the IDL and the generation of corresponding service stubs

1. A Transport Layer (TCP, UDP, HTTP, etc..)
2. Message encoding/decoding Layer

1. Define the message structure and protocol to be agnostic of transport used.
2. Evaluate different Message Serialization Frameworks and RPC Frameworks.

Message Structure Definition

All details relating to the Message structure definition can be found here.

Connect 

In order to fit into the existing Geode client/server infrastructure, we will be leveraging the current Geode "cache server" component.  It accepts an initial byte that tells it what type of client is connecting to the server and how the client should be handled. A client using the new protocol can connect with the Geode server by sending a protocol byte. Initially, we will support the following two protocols:

• byte - 110: Message will contain the whole request or response. 
• byte - 111: If the message is large then client or server can divide the message into the set of small messages. Then they need to collect all the small message and parse the whole request or response.

We may add additional protocol definitions here to indicate the use of an alternative serialization mechanism.  For instance bytes 112 & 113 might indicate the but require the server to use Protobuf for serialization of client/server messages.  How we handle other serialization libraries will be addressed when the pluggable-serialization architecture is roughed out.

Protocol FieldDefinitions

API Id

The ApiId recognizes the API, a client wants to invoke on the server. The request format will contain the 2-byte(int16) for API id. It will be marked as ApiId in the request format.

API Version

API version will be associated with the request API. The request format will contain a 1-byte(int8) for the version. It will be marked as ApiVersion in the request format. Its current value will be 1.

Correlation Id

The purpose of correlation id to match the request and its corresponding response. The request format will contain the 4-bytes(int32) for correlation Id. It will be marked as CorrelationId in the request and response format. The client needs to send correlation id with the request and server will send the same id with the response.

Object Type

We will support all the object types which Geode understands. This would include all the primitive java types, an array of primitive types, collections, java serialization, data serializable, PDX serialization and custom user data serializers. The client needs to serialize objects as described here.  This might be extended as part of making the Geode storage format pluggable.

ObjectKeyType

Geode supports only few object types as the region Key. The region key will be marked as Key in the request format. the client needs to serialize key as described here.

ResponseType

The Response Type indicates whether the response is partial or complete. The response with the FullResponse type id indicates the completion of that request. And the response with PartialResponse type id indicates the response is partial. The response format will contain the 2-bytes(int16) for response type. It will be marked as ResponseTypeId in the response format.

Error Codes

The Error codes indicate the issue with the invocation of API at the server. We have defined the error code for various issues here. The response format will contain the 2-bytes(int16) for error codes. It will be marked as ErrorCode in the response format.

Data Type Definitions

The protocol descriptions use the following types. These may be mapped to a pluggable serialization description language but their native serialization is described here.  The native serialization uses network byte order ("big-endian").

Message Definition

A message is a series of bytes which contains the request or response. If the message is large, then we will have provision to divide the message into small messages. In that case, client/server needs to collect all messages to parse the request/response.

The message will be sent in following way.  A client can send the multiple messages on the connection and the server will respond to those messages in same order. 

Additional Request and Response message definitions can be found in the API's section.

Message definition grammar

In order to consistently define messages the Extended Backus–Naur form grammar will be used.

Generic Message definition

Every message will adhere to the following generic message definition. A Message will comprise of a MessageHeader and either a Request or Response component.

MessageHeader

The Message header is a fixed size header which contains the size of a message, boolean flag to indicates whether a message is partial, and correlation id for that request message. The correlation id is used for the dual purpose here.

• If a message is sent in multiple sub-messages, then it will be used for combining the whole message. 
• It will be used for correlating the request to its response.

Request

The request would contain the fixed size request header, optional metadata and request API parameters. 

RequestHeader

The request header contains the ApiId, ApiVersion, and hasMetaData flag to indicate whether the request contains some metadata.

Response

The response would contain the fixed size response header, optional metadata and return values.

ResponseHeader

The response header will have resposneType, which indicates its partial response, full response or error. A hasMetaData flag indicates whether the response contains some metadata.

ErrorResponse

The server will raise the error when it failed to execute API request from the client. The error code and message should help the client to diagnose the issue.

Value

The Value is serialized bytes for the Geode region value. It contains value header and series of bytes. Using value header, we can send a big serialized object in more than one chunk. 

ValueHeader

The value header contains the value bytes size, and a flag indicates whether it contains all the value bytes. 

MetaData

The purpose of a metadata to pass defined key value pair with request and response. That will be optional for a client. If there is any metadata associated with request or response, then need to set "hasMetadata" flag to "true" in request or response header. After that send metadata in the following format.

Supported MetaData

 We would have following pre-defined key and value for a metadata. Note this list will grow over time.

MetaData for Requests

MetaData for Responses

Examples

PutRequest

string regionName = "ExampleRegion"

Key = 101

Value = "New Geode Client Server Protocol"

CallbackArg = Null

PutResponse

Messages

PutRequestMessage {
	MessageHeader {
  		Size, 4 byte, int32
 		CorrelationId, 4 byte, int32
        isPartialMessage, 1 byte, boolean
        hasMetaData, 1 byte, boolean
 	}
 
 	RequestHeader {
 		ApiId, 2 byte, int16
 		ApiVersion, 1 byte, int8
 	}
 	PutRequest {
 		regionName, varaible , String {
			len, 2 byte, int16
			variable, utf encoding		
		}
 		key, variable, bytes
		CallbackArg, variable, bytes
 		Value {
 			ValueHeader {
				Size, 4 byte, int32
				hasPartialBytes, 1 byte, boolean
 			}
			value {
 				bytes, series of bytes
			}
 		}
 	}
}

PutResponseMessage {
	MessageHeader {
    	Size, 4 byte, int32
        CorrelationId, 4 byte, int32
        isPartialMessage, 1 byte, boolean
        hasMetaData, 1 byte, boolean
    }
    ResponseHeader {
 		ResponseTypeId, 2 byte, int16	
 	}
  	Success, 1 byte, boolean
}

GetRequestMessage {
	MessageHeader {
  		Size, 4 byte, int32
 		CorrelationId, 4 byte, int32
        isPartialMessage, 1 byte, boolean
        hasMetaData, 1 byte, boolean
 	}
 
 	RequestHeader {
 		ApiId, 2 byte, int16
 		ApiVersion, 1 byte, int8
 	}
 	GetRequest {
 		regionName, varaible , String {
			len, 2 byte, int16
			variable, utf encoding		
		}
 		key, variable, bytes
		CallbackArg, variable, bytes
 	}
}

GetResponseMessage {
	MessageHeader {
    	Size, 4 byte, int32
        CorrelationId, 4 byte, int32
        isPartialMessage, 1 byte, boolean
        hasMetaData, 1 byte, boolean
    }
    ResponseHeader {
 		ResponseTypeId, 2 byte, int16
 	}
  	Result, variable, bytes
}

PutAllRequestMessage {
	MessageHeader {
  		Size, 4 byte, int32
 		CorrelationId, 4 byte, int32
        isPartialMessage, 1 byte, boolean
        hasMetaData, 1 byte, boolean
 	}
 
 	RequestHeader {
 		ApiId, 2 byte, int16
 		ApiVersion, 1 byte, int8
 	}
 	PutRequest {
 		regionName, varaible , String {
			len, 2 byte, int16
			variable, utf encoding		
		}
		NumberOfKeyValuePair, 4 byte, int32 
		KeyValuePair  { 
	 		key, variable, bytes		
 			Value {
 				ValueHeader {
					Size, 4 byte, int32
					hasPartialBytes, 1 byte, boolean
 				}
				value {
 					bytes, series of bytes
				}
			}
 		}
		CallbackArg, variable, bytes
 	}
}

PutAllResponseMessage {
	MessageHeader {
    	Size, 4 byte, int32
        CorrelationId, 4 byte, int32
        isPartialMessage, 1 byte, boolean
        hasMetaData, 1 byte, boolean
    }
	ResponseHeader {
 		ResponseTypeId, 2 byte, int16
 	}
  	NumberOfKeysFailed , 4 byte, int32
}

GetAllRequestMessage {
	MessageHeader {
  		Size, 4 byte, int32
		CorrelationId, 4 byte, int32
        isPartialMessage, 1 byte, boolean
        hasMetaData, 1 byte, boolean
 	}
 
 	RequestHeader {
 		ApiId, 2 byte, int16
 		ApiVersion, 1 byte, int8
 	}
 	GetRequest {
 		regionName, varaible , String {
			len, 2 byte, int16
			variable, utf encoding		
		}
		NumberOfKeys, 4 byte, int32 
		keys {
 			key, variable, bytes
		}
		CallbackArg, variable, bytes
 	}
}

GetAllResponseMessage {
	MessageHeader {
    	Size, 4 byte, int32
		CorrelationId, 4 byte, int32
        isPartialMessage, 1 byte, boolean
        hasMetaData, 1 byte, boolean
    }
    ResponseHeader {
 		ResponseTypeId, 2 byte, int16
 	}
  	NumberOfKeyValuePair, 4 byte, int32 
	KeyValuePair  { 
		key, variable, bytes		
 		value, variable, bytes
	}
}

RPC Frameworks

RPC frameworks such as Thrift or Apache Avro provide tools to generate client -server library based on a message schema. We are talking about how we may support various RPC frameworks to facilitate quick creation of GEODE clients in various languages supported by popular RPC frameworks.

Glossary

• Partial Response
• Full Response