Introduction
Apache Geode is a data management platform that provides real-time, consistent access to data-intensive applications throughout widely distributed cloud architectures. While it currently has high-speed client interfaces for Java, C++ and .NET there is both a need to create lighter-weight clients and a demand to access Geode from other programming languages. Unfortunately, the existing client-server protocol is undocumented. It evolved over time and is overly complex to meet either of these needs.
This document describes a new, lighter-weight client/server protocol that may be implemented in phases, moving from a "basic client" to a more advanced "smart client". It endeavors to provide a protocol that is also more amenable to more modern APIs such as those using asynchronous or reactive patterns.
You will notice that this document describes the protocol down to the level of byte-ordering and "bytes on the wire". This is a description of the native serialization of the protocol that will be used for the initial implementation. It is our intent to make this pluggable so that alternative serialization technologies may be used instead, such as Protobuf or Apache Thrift. How that is done will be left as a goal for the architecture of the server component, with an additional goal of providing an IDL description of the protocol.
At the same time, we realize that serialization of application keys, values, callback arguments, function parameters and so forth are a separate matter and are not necessarily tied to the serialization protocol used for client/server messaging. The initial protocol will support primitive types such as scalars, strings, and byte arrays. It will also support JSON documents as values and convert between these and Geode PDX-serialized objects in the servers.
Goals
The high-level goals for the protocol are defined here.
Protocol Requirements
In the evaluation or definition of any protocol, it expected that the evaluated protocol/framework meets the following requirements:
Versioning: The protocol has to provide version information, in order to distinguish different protocol versions from one another.
Correlation Id: This number is a unique identifier that allows the system to correlate requests and responses.
Object Type: The type of a serialized object.
Response Type: It indicates whether a response is partial or complete.
ErrorCodes: It indicates the problem with API invocation.
Chunk Response: The ability to send a large response in multiple smaller, more manageable chunks.
Continuous Response: Client can register(Observer pattern) for events and then server notify the client if those events occur.
Request: The request message to be sent
Response: The response message received in relation to a request message
Request Format: Format of request API and its parameters, which client wants to invoke.
Response Format: Format for API return value, which client invoked.
Message: The generic construct that represents a message that is to be sent which contains a Message Header and Request/Response.
Serialized Byte Order: Big Endian
Approach
During the investigation into frameworks that would "lower the barrier of entry," it became evident that there are two approaches that can be taken to solve the problem:
- Message Serialization Frameworks - These frameworks allow for the definition of a message in a generic IDL, the generation of language specific classes from the IDL, and the encoding/decoding of those message to be sent over a transport
- RPC Frameworks - There frameworks provide greater coverage in the node-to-node communication: the transport layer (HTTP, TCP, UDP), the message definition in IDL with corresponding serialization mechanism and the definition of methods in the IDL and the generation of corresponding service stubs
- A Transport Layer (TCP, UDP, HTTP, etc..)
- Message encoding/decoding Layer
- Define the message structure and protocol to be agnostic of transport used.
- Evaluate different Message Serialization Frameworks and RPC Frameworks.
Message Structure Definition
All details relating to the Message structure definition can be found on the page Message Structure and Definition.
RPC Framework Evaluation
RPC frameworks such as Apache Thrift, gRPC or Apache Avro provide tools to generate client -server library based on a message schema. We are talking about how we may support various RPC frameworks to facilitate quick creation of GEODE clients in various languages supported by popular RPC frameworks.
The evaluation of each of the RPC frameworks can be found on the page RPC framework evaluation
Examples
PutRequest
string regionName = "ExampleRegion"
Key = 101
Value = "New Geode Client Server Protocol"
CallbackArg = Null
MessageHeader
| RequestHeader | PutRequest | |
---|---|---|---|
Size CorrelationId PartialMessage hasMetaData | RequestType apiVersion | RegionName Key CallbackArg | Value ( ValueHeader value ) |
Size = Size of Request (65) 0x00 0x00 0x00 0x42 | RequestType (PutRequestType = 3) 0x00 0x03 | RegionName(type:String, value:"ExampleRegion" ) len = 0x00 0x0d Utf Encoding = 0x45 0x78 0x61 0x6d 0x70 0x6c 0x65 0x52 0x65 0x67 0x69 0x6f 0x6e | Size = (number of serialized bytes = 35) 0x00 0x00 0x00 0x23 |
CorrelationId = 1 0x00 0x00 0x00 0x01 | apiVersion (1) 0x01 | Key (Serialzied using geode types, value = 101) Geode Int type = 0x39 Value = 0x00 0x00 0x00 0x65 | isPartialBytes = (It contains all serialized bytes, type = boolean) 0x00 |
PartialMessage = (type = Boolean, value = false) 0x00 |
| CallbackArg (Serialzied using geode types, value = null) Value = 0x29 | value (Serialized as Geode String type, value = "New Geode Client Server Protocol") Geode String type = 0x57 Serialized Encoded length = 0x00 0x20 Encoded String = 0x4e 0x65 0x77 0x20 0x47 0x65 0x6f 0x64 0x65 0x20 0x43 0x6c 0x69 0x65 0x6e 0x74 0x20 0x53 0x65 0x72 0x76 0x65 0x72 0x20 0x50 0x72 0x6f 0x74 0x6f 0x63 0x6f 0x6c |
hasMetaData (false) 0x00 |
PutResponse
MessageHeader | ResponseHeader | PutResponse |
---|---|---|
Size CorrelationId PartialMessage hasMetaData | ResponseTypeId | Success |
Size = Size of Request (4) 0x00 0x00 0x00 0x04 | ResponseTypeId (FullResponse, type=int16, value =1) 0x00 0x01 | Success(type=boolean, value = true) 0x01 |
CorrelationId = 1 0x00 0x00 0x00 0x01 |
| |
PartialMessage = (type = Boolean, value = false) 0x00 | ||
hasMetaData (false) 0x00 |
Messages
PutRequestMessage | PutResponseMessage |
---|---|
PutRequestMessage { MessageHeader { Size, 4 byte, int32 CorrelationId, 4 byte, int32 isPartialMessage, 1 byte, boolean hasMetaData, 1 byte, boolean } RequestHeader { ApiId, 2 byte, int16 ApiVersion, 1 byte, int8 } PutRequest { regionName, varaible , String { len, 2 byte, int16 variable, utf encoding } key, variable, bytes CallbackArg, variable, bytes Value { ValueHeader { Size, 4 byte, int32 hasPartialBytes, 1 byte, boolean } value { bytes, series of bytes } } } } | PutResponseMessage { MessageHeader { Size, 4 byte, int32 CorrelationId, 4 byte, int32 isPartialMessage, 1 byte, boolean hasMetaData, 1 byte, boolean } ResponseHeader { ResponseTypeId, 2 byte, int16 } Success, 1 byte, boolean } |
GetRequestMessage | GetResponseMessage |
GetRequestMessage { MessageHeader { Size, 4 byte, int32 CorrelationId, 4 byte, int32 isPartialMessage, 1 byte, boolean hasMetaData, 1 byte, boolean } RequestHeader { ApiId, 2 byte, int16 ApiVersion, 1 byte, int8 } GetRequest { regionName, varaible , String { len, 2 byte, int16 variable, utf encoding } key, variable, bytes CallbackArg, variable, bytes } } | GetResponseMessage { MessageHeader { Size, 4 byte, int32 CorrelationId, 4 byte, int32 isPartialMessage, 1 byte, boolean hasMetaData, 1 byte, boolean } ResponseHeader { ResponseTypeId, 2 byte, int16 } Result, variable, bytes } |
PutAllRequestMessage | PutAllResponseMessage |
PutAllRequestMessage { MessageHeader { Size, 4 byte, int32 CorrelationId, 4 byte, int32 isPartialMessage, 1 byte, boolean hasMetaData, 1 byte, boolean } RequestHeader { ApiId, 2 byte, int16 ApiVersion, 1 byte, int8 } PutRequest { regionName, varaible , String { len, 2 byte, int16 variable, utf encoding } NumberOfKeyValuePair, 4 byte, int32 KeyValuePair { key, variable, bytes Value { ValueHeader { Size, 4 byte, int32 hasPartialBytes, 1 byte, boolean } value { bytes, series of bytes } } } CallbackArg, variable, bytes } } | PutAllResponseMessage { MessageHeader { Size, 4 byte, int32 CorrelationId, 4 byte, int32 isPartialMessage, 1 byte, boolean hasMetaData, 1 byte, boolean } ResponseHeader { ResponseTypeId, 2 byte, int16 } NumberOfKeysFailed , 4 byte, int32 } |
GetAllRequestMessage | GetAllResponseMessage |
GetAllRequestMessage { MessageHeader { Size, 4 byte, int32 CorrelationId, 4 byte, int32 isPartialMessage, 1 byte, boolean hasMetaData, 1 byte, boolean } RequestHeader { ApiId, 2 byte, int16 ApiVersion, 1 byte, int8 } GetRequest { regionName, varaible , String { len, 2 byte, int16 variable, utf encoding } NumberOfKeys, 4 byte, int32 keys { key, variable, bytes } CallbackArg, variable, bytes } } | GetAllResponseMessage { MessageHeader { Size, 4 byte, int32 CorrelationId, 4 byte, int32 isPartialMessage, 1 byte, boolean hasMetaData, 1 byte, boolean } ResponseHeader { ResponseTypeId, 2 byte, int16 } NumberOfKeyValuePair, 4 byte, int32 KeyValuePair { key, variable, bytes value, variable, bytes } } |
Glossary
- Partial Response
- Full Response