Introduction

Apache Geode is a data management platform that provides real-time, consistent access to data-intensive applications throughout widely distributed cloud architectures. While it currently has high-speed client interfaces for Java, C++ and .NET there is both a need to create lighter-weight clients and a demand to access Geode from other programming languages. Unfortunately, the existing client-server protocol is undocumented.  It evolved over time and is overly complex to meet either of these needs.

This document describes a new, lighter-weight client/server protocol that may be implemented in phases, moving from a "basic client" to a more advanced "smart client".  It endeavors to provide a protocol that is also more amenable to more modern APIs such as those using asynchronous or reactive patterns.

You will notice that this document describes the protocol down to the level of byte-ordering and "bytes on the wire".  This is a description of the native serialization of the protocol that will be used for the initial implementation.  It is our intent to make this pluggable so that alternative serialization technologies may be used instead, such as Protobuf or Apache Thrift.  How that is done will be left as a goal for the architecture of the server component, with an additional goal of providing an IDL description of the protocol.

At the same time, we realize that serialization of application keys, values, callback arguments, function parameters and so forth are a separate matter and are not necessarily tied to the serialization protocol used for client/server messaging.  The initial protocol will support primitive types such as scalars, strings, and byte arrays.  It will also support JSON documents as values and convert between these and Geode PDX-serialized objects in the servers.

Goals

The high-level goals for the protocol are defined here.

Approach

 During the investigation into frameworks that would "lower the barrier of entry," it became evident that there are two approaches that can be taken to solve the problem:

1. Message Serialization Frameworks - These frameworks allow for the definition of a message in a generic IDL, the generation of language specific classes from the IDL, and the encoding/decoding of those message to be sent over a transport
2. RPC Frameworks - There frameworks provide greater coverage in the node-to-node communication: the transport layer (HTTP, TCP, UDP), the message definition in IDL with corresponding serialization mechanism and the definition of methods in the IDL and the generation of corresponding service stubs

1. A Transport Layer (TCP, UDP, HTTP, etc..)
2. Message encoding/decoding Layer

1. Define the message structure and protocol to be agnostic of transport used.
2. Evaluate different Message Serialization Frameworks and RPC Frameworks.

Message Structure Definition

All details relating to the Message structure definition can be found on the page Message Structure and Definition.

Examples

PutRequest

string regionName = "ExampleRegion"

Key = 101

Value = "New Geode Client Server Protocol"

CallbackArg = Null

PutResponse

Messages

PutRequestMessage {
	MessageHeader {
  		Size, 4 byte, int32
 		CorrelationId, 4 byte, int32
        isPartialMessage, 1 byte, boolean
        hasMetaData, 1 byte, boolean
 	}
 
 	RequestHeader {
 		ApiId, 2 byte, int16
 		ApiVersion, 1 byte, int8
 	}
 	PutRequest {
 		regionName, varaible , String {
			len, 2 byte, int16
			variable, utf encoding		
		}
 		key, variable, bytes
		CallbackArg, variable, bytes
 		Value {
 			ValueHeader {
				Size, 4 byte, int32
				hasPartialBytes, 1 byte, boolean
 			}
			value {
 				bytes, series of bytes
			}
 		}
 	}
}

PutResponseMessage {
	MessageHeader {
    	Size, 4 byte, int32
        CorrelationId, 4 byte, int32
        isPartialMessage, 1 byte, boolean
        hasMetaData, 1 byte, boolean
    }
    ResponseHeader {
 		ResponseTypeId, 2 byte, int16	
 	}
  	Success, 1 byte, boolean
}

GetRequestMessage {
	MessageHeader {
  		Size, 4 byte, int32
 		CorrelationId, 4 byte, int32
        isPartialMessage, 1 byte, boolean
        hasMetaData, 1 byte, boolean
 	}
 
 	RequestHeader {
 		ApiId, 2 byte, int16
 		ApiVersion, 1 byte, int8
 	}
 	GetRequest {
 		regionName, varaible , String {
			len, 2 byte, int16
			variable, utf encoding		
		}
 		key, variable, bytes
		CallbackArg, variable, bytes
 	}
}

GetResponseMessage {
	MessageHeader {
    	Size, 4 byte, int32
        CorrelationId, 4 byte, int32
        isPartialMessage, 1 byte, boolean
        hasMetaData, 1 byte, boolean
    }
    ResponseHeader {
 		ResponseTypeId, 2 byte, int16
 	}
  	Result, variable, bytes
}

PutAllRequestMessage {
	MessageHeader {
  		Size, 4 byte, int32
 		CorrelationId, 4 byte, int32
        isPartialMessage, 1 byte, boolean
        hasMetaData, 1 byte, boolean
 	}
 
 	RequestHeader {
 		ApiId, 2 byte, int16
 		ApiVersion, 1 byte, int8
 	}
 	PutRequest {
 		regionName, varaible , String {
			len, 2 byte, int16
			variable, utf encoding		
		}
		NumberOfKeyValuePair, 4 byte, int32 
		KeyValuePair  { 
	 		key, variable, bytes		
 			Value {
 				ValueHeader {
					Size, 4 byte, int32
					hasPartialBytes, 1 byte, boolean
 				}
				value {
 					bytes, series of bytes
				}
			}
 		}
		CallbackArg, variable, bytes
 	}
}

PutAllResponseMessage {
	MessageHeader {
    	Size, 4 byte, int32
        CorrelationId, 4 byte, int32
        isPartialMessage, 1 byte, boolean
        hasMetaData, 1 byte, boolean
    }
	ResponseHeader {
 		ResponseTypeId, 2 byte, int16
 	}
  	NumberOfKeysFailed , 4 byte, int32
}

GetAllRequestMessage {
	MessageHeader {
  		Size, 4 byte, int32
		CorrelationId, 4 byte, int32
        isPartialMessage, 1 byte, boolean
        hasMetaData, 1 byte, boolean
 	}
 
 	RequestHeader {
 		ApiId, 2 byte, int16
 		ApiVersion, 1 byte, int8
 	}
 	GetRequest {
 		regionName, varaible , String {
			len, 2 byte, int16
			variable, utf encoding		
		}
		NumberOfKeys, 4 byte, int32 
		keys {
 			key, variable, bytes
		}
		CallbackArg, variable, bytes
 	}
}

GetAllResponseMessage {
	MessageHeader {
    	Size, 4 byte, int32
		CorrelationId, 4 byte, int32
        isPartialMessage, 1 byte, boolean
        hasMetaData, 1 byte, boolean
    }
    ResponseHeader {
 		ResponseTypeId, 2 byte, int16
 	}
  	NumberOfKeyValuePair, 4 byte, int32 
	KeyValuePair  { 
		key, variable, bytes		
 		value, variable, bytes
	}
}

RPC Framework Evaluation

RPC frameworks such as Thrift or Apache Avro provide tools to generate client -server library based on a message schema. We are talking about how we may support various RPC frameworks to facilitate quick creation of GEODE clients in various languages supported by popular RPC frameworks.

Glossary

• Partial Response
• Full Response