You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 102 Next »

 

 

Introduction

Apache Geode is a data management platform that provides real-time, consistent access to data-intensive applications throughout widely distributed cloud architectures. While it currently has high-speed client interfaces for Java, C++ and .NET there is both a need to create lighter-weight clients and a demand to access Geode from other programming languages. Unfortunately, the existing client-server protocol is undocumented.  It evolved over time and is overly complex to meet either of these needs.

This document describes a new, lighter-weight client/server protocol that may be implemented in phases, moving from a "basic client" to a more advanced "smart client".  It endeavors to provide a protocol that is also more amenable to more modern APIs such as those using asynchronous or reactive patterns.

You will notice that this document describes the protocol down to the level of byte-ordering and "bytes on the wire".  This is a description of the native serialization of the protocol that will be used for the initial implementation.  It is our intent to make this pluggable so that alternative serialization technologies may be used instead, such as Protobuf or Apache Thrift.  How that is done will be left as a goal for the architecture of the server component, with an additional goal of providing an IDL description of the protocol.

At the same time, we realize that serialization of application keys, values, callback arguments, function parameters and so forth are a separate matter and are not necessarily tied to the serialization protocol used for client/server messaging.  The initial protocol will support primitive types such as scalars, strings, and byte-arrays.  It will also support JSON documents as values and convert between these and Geode PDX-serialized objects in the servers.

Goals

The high-Level Goals for the protocol are defined here.


Protocol Terms

  Any binary protocol requires the following things:

  1. Version: This indicates the API version.

  2. Correlation Id: This should be different per request sent. It allows correlation of request and response.

  3. Object Type: The type of a serialized object.

  4. Response Type: It indicates whether a response is partial or complete.

  5. ErrorCodes: It indicates the problem with API invocation.

  6. Chunk Response: Send large response in multiple chunks.

  7. Continuous Response: Client can register(Observer pattern) for events and then server notify the client if those events occur.

  8. Request: It indicates client's message

  9. Response: It indicates server's message.

  10. Request Format: Format of request API and its parameters, which client wants to invoke.

  11. Response Format: Format for API return value, which client invoked.

  12. Message: Set of bytes which contain the Message Header and Request/Response.

  13. Serialized Byte Order: Big Endian

Connect 

In order to fit into the existing Geode client/server infrastructure, we will be leveraging the current Geode "cache server" component.  It accepts an initial byte that tells it what type of client is connecting to the server and how the client should be handled. A client using the new protocol can connect with the Geode server by sending a protocol byte. Initially, we will support the following two protocols:

  • byte - 110: Message will contain the whole request or response. 
  • byte - 111: If the message is large then client or server can divide the message into the set of small messages. Then they need to collect all the small message and parse the whole request or response.

We may add additional protocol definitions here to indicate use of an alternative serialization mechanism.  For instance bytes 112 & 113 might indicate the but require the server to use protobuf for serialization of client/server messages.  How we handle other serialization libraries will be addressed when the pluggable-serialization architecture is roughed out.

API Id

The ApiId recognizes the API, a client wants to invoke on the server. The request format will contain the 2-byte(int16) for Api id. It will be marked as ApiId in the request format.

API Version

API version will be associated with the request api. The request format will contain a 1-byte(int8) for the version. It will be marked as ApiVersion in the request format. Its current value will be 1.

Correlation Id

The purpose of correlation id to match the request and its corresponding response. The request format will contain the 4-bytes(int32) for correlation Id. It will be marked as CorrelationId in the request and response format. The client needs to send correlation id with the request and server will send the same id with the response.

Object Type

We will support all the object types which Geode understands. This would include all the primitive java types, an array of primitive types, collections, java serialization, data serializable, pdx serialization and custom user data serializers. The client needs to serialize objects as described here.  This might be extended as part of making the Geode storage format pluggable.

ObjectKeyType

Geode supports only few object types as the region Key. The region key will be marked as Key in the request format. the client needs to serialize key as described here.

ResponseType

The Response Type indicates whether the response is partial or complete. The response with the FullResponse type id indicates the completion of that request. And the response with PartialResponse type id indicates the response is partial. The response format will contain the 2-bytes(int16) for response type. It will be marked as ResponseTypeId in the response format.

ResponseTypeResponseTypeId
FullResponse1
PartialResponse2

Error Codes

The Error codes indicate the issue with the invocation of API at the server. We have defined the error code for various issues hereThe response format will contain the 2-bytes(int16) for error codes. It will be marked as ErrorCode in the response format.

Format grammar

The format is described using Extended Backus–Naur form grammar.

UsageNotation
definition=
alteration|
optional[ ... ]
repetition{ ... }

ProtocolType

The protocol descriptions use the following types. These may be mapped to a pluggable serialization description language but their native serialization is described here.  The native serialization uses network byte-order ("big-endian").

 

TypeNumber Of BytesValueSerializedBytes
booleanFixed = 1true0x01 
booleanFixed = 1false0x00
int8Fixed = 110x01
int16Fixed = 210x00 0x01 
int32Fixed = 410x00 0x00 0x00 0x01 
int64Fixed = 810x00 0x00 0x00 0x00 0x00 0x00 0x00 0x01 
String(modified UTF 8 )

Variable

  • 2 byte(int16) for length of encoded String
  • Utf Encoding
"Geode"

0x00 0x05 (length)

0x47 0x65 0x6f 0x64 0x65 (utf encoding)

byte[]

Variable

  • 2 bytes(int16) for number of bytes
  • series of bytes.
{1,2}

0x00 0x02 (length)

0x01 0x02 

bytes

Variable: series of bytes containing a serialized value.

  


Message(Framing) 

A message is a series of bytes which contains the request or response. If the message is large, then we will have provision to divide the message into small messages. In that case, client/server needs to collect all messages to parse the request/response. The message will be sent in following way.  A client can send the multiple messages on the connection and the server will respond to those messages in same order. 

 

Message => MessageHeader (Request | Response )
MessageHeader => defined below
Request => defined below
Response => defined below

MessageHeader

The Message header is a fixed size header which contains the size of a message, boolean flag to indicates whether a message is partial, and correlation id for that request message. The correlation id is used for the dual purpose here.

  • If a message is sent in multiple sub-messages, then it will be used for combining the whole message. 
  • It will be used for correlating the request to its response.
MessageHeader => Size isPartialMessage CorrelationIdDescription
Size => fixedSize = 4 bytes, type = int32Size of request or response
isPartialMessage => fixedSize = 1 byte, type = boolean 
CorrelationId => fixedSize = 4 bytes, type = int32 

Request Format

The request would contain the fixed size request header, optional metadata and request api parameters. 

Request => RequestHeader [MetaData] RequestAPI
RequestHeader => defined below
MetaData => optional

RequestAPI => (PutRequest | GetRequest | PutAllRequest | GetAllRequest |ServerConfigRequest | ClientConfigRequest | AuthRequest)

RequestHeader

The request header contains the ApiId, ApiVersion, and hasMetaData flag to indicate whether the request contains some metadata.

RequestHeader => ApiId apiVersion hasMetaDataDescription
ApiId => fixedSize = 2 bytes, type = int16 
apiVersion => fixedSize = 1 byte, type = int8 
hasMetaData => fixedSize = 1 byte, type = booleanif there is any meta data associated with this request

Response Format

The response would contain the fixed size response header, optional metadata and return values.

  

Response => ResponseHeader [MetaData] APIResponse
ResponseHeader => defined below
MetaData => Optional
APIResponse => (PutResponse | GetResponse | PutAllResponse | GetAllResponse | ServerConfigResposne | ClientConfigResponse | AuthResponse | ErrorResponse)

ResponseHeader

The response header will have resposneType, which indicates its partial response, full response or error. A hasMetaData flag indicates whether the response contains some metadata.

ResponseHeader => (ResponseTypeId | ErrorCode) hasMetaData CorrelationIdDescription
ResponseTypeId => fixedSize = 2 bytes, type = int16 
ErrorCode => fixedSize = 2 bytes, type = int16When there is error response will have error message(String) for it
hasMetaData => fixedSize = 1 byte, type = booleanif there is any meta data associated with this request

ErrorResponse

The server will raise the error when it failed to execute api request from the client. The error code and message should help the client to diagnose the issue.

ErrorResponse => errorMessage
errorMessage => (variable size, type = String)

Value

The Value is serialized bytes for the Geode region value. It contains value header and series of bytes. Using value header, we can send a big serialized object in more than one chunk. 

Value => ValueHeader value {ValueHeader value}Description
ValueHeader => defined below 
value => bytesSerialized Value Object which Geode can understand

ValueHeader

The value header contains the value bytes size, and a flag indicates whether it contains all the value bytes. 

ValueHeader => Size hasPartialBytesDescription
Size => fixedSize = 4 bytes, type = int32Number of serialized bytes
hasPartialBytes => fixedSize = 1 byte, type = booleanWhether this contains partial bytes of value

APIS

PutRequest

PutRequest => RegionName Key CallbackArg Value
RegionName => (variable size, type = String)
Key => bytes
CallbackArg =>  bytes 
Value => defined above

PutResponse

PutResponse => Success
Success => fixedSize = 1 byte, type = boolean

GetRequest

GetRequest => RegionName Key CallbackArg
RegionName => (variable size, type = String)
Key => bytes
CallbackArg => bytes

GetResponse

GetResponse => Result
Result => bytes

GetAllRequest

GetAllRequest => RegionName NumberOfKeys Key {Key} CallbackArg
RegionName => (variable size, type = String)
NumberOfKeys => fixedSize = 4 bytes, type = int32
Key => bytes
CallbackArg => bytes

GetAllResponse

GetAllResponse => NumberOfKeyValuePair Key value {Key value}
NumberOfKeyValuePair => fixedSize = 4 bytes, type = int32
Key => bytes
value => bytes

PutAllRequest

PutAllRequest => RegionName NumberOfKeys Key Value{Key Value} CallbackArg
RegionName => (variable size, type = String)
NumberOfKeys => fixedSize = 4 bytes, type = int32
Key => bytes
Value => defined above
CallbackArg => bytes

PutAllResponse

PutAllResponse => NumberOfKeysFailed {Key}Description
NumberOfKeysFailed => fixedSize = 4 bytes, type = int32In general this will be 0. But if Geode unable to put some keys in Region then client can retry those keys.
Key => bytes 

Metadata

The purpose of a metadata to pass defined key value pair with request and response. That will be optional for client. If there is any metadata associated with request or response, then need to set "hasMetadata" flag to "true" in request or response header. After that send metadata in following format.

MetaData => NumberOfMetadata MetadataKeyId MetadataKeyValue { MetadataKeyId MetadataKeyValue}
NumberOfMetadata=> fixedSize = 2 bytes, type = int16
MetadataKeyId => fixedSize = 2 bytes, type = int16
MetadataKeyValue => variable, Value as defined in table below


We would have following pre-defined key and value for a metadata. Note this list will grow over time.

Request MetaData KeyMetaData KeyIdMetaData ValueDescription
JSON_KEY

1

fixedSize = 2 bytes, type = int16

true

fixedSize = 1 byte, type = boolean

Geode will expect key as JSON string(or bytes) and it will convert that string into pdx key.

If the response requires a key, then it will convert the pdx key to JSON string(or bytes) back.

JSON_VALUE

2

fixedSize = 2 bytes, type = int16

true

fixedSize = 1 byte, type = boolean

Geode will expect Value as JSON string(or bytes) and it will convert that string into pdx value.

If the response requires a value, then it will convert pdx value to JSON string(or bytes) back.

EVENT_ID

3

fixedSize = 2 bytes, type = int16

EventId {

uniqueId: type = String

ThreadId:type=int64

SequenceId: type=int64

}

The eventid is used to identify same region event in Geode. Geode keeps map of "uniqueId + threadId" Vs

"SequenceId" to know whether region event has been already seen or not.

 

Response MetaData KeyMetaData KeyIdMetaData ValueDescription
UPDATE_PR_META_DATA

1

fixedSize = 2 bytes, type = int16

true

fixedSize = 1 byte, type = boolean

[optional]The server accepted and forwarded the request to the appropriate

node holding the affected cache entry.  A smart client should refresh its partitioned

region location metadata for higher performance.

ServerConfigRequest

The purpose of ServerConfigRequest request to get a server config from the server. The client needs to send this request after connecting to the server. If the client knows server properties, then there is no need to send this request.

 

ServerConfigRequest => {}
ServerConfigResponse =>NumberOfProperties PropertyId PropertyValue{ PropertyId PropertyValue}
NumberOfProperties => fixedSize = 2 bytes, type = int16
Propertyid => fixedSize = 2 bytes, type = int16
PropertyValue => variable, Value as defined in table below
Server Response PropertiesPropertyIdPropertyValueDescription
SECURITY_ENABLED1booleanWhether security is enabled at server
DIFFIE_HELLMAN_KEY2byte[]The server Diffie-Hellman key if the credential is required to encrypt.
MAX_TIME_BETWEEN_CLIENT_PING3int32If the client connection is idle for MAX_TIME_BETWEEN_CLIENT_PING seconds then the server will close that connection.


ClientConfigRequest

The Purpose of ClientConfigRequest request to send a client config to a server. The client needs to send this request after connecting to the server. This request is optional for a client unless a server is configured for the Diffie-hellman algorithm.

ClientConfigRequestNumberOfProperties PropertyId PropertyValue{ PropertyId PropertyValue}
NumberOfPropertiesfixedSize = 2 bytes, type = int16
PropertyidfixedSize = 2 bytes, type = int16
PropertyValuevariable, Value as defined in table below
ClientConfigResponse ==> Success
Success => fixedSize = 1 byte, type = boolean
Client Request PropertiesPropertyIdPropertyValueDescription 
CLIENT_ID1string

Unique id for the client; if a client doesn't send this property then the server

will just create id (client-host, client-port, server-host, server-port).

optional
ClIENT_READ_TIMEOUT2int32

The client will wait for ClIENT_READ_TIMEOUT for a server response. It's an optional

property.

optional
DIFFIE_HELLMAN_KEY3byte[]

The client Diffie-Hellman key is required to encrypt the credential.

If a server is configured with it then the client should send this.

AuthRequest

The purpose of AuthRequest to authenticate the client connection. A client can send the auth request in the following format. Before sending the auth request a client can verify the server config by sending the ServerConfigRequest. A client can send a key-value pair of credentials to authenticate itself.

Those pair needs to serialized in following way. If Diffie-Hellman encryption is enabled on the server side then need to encrypt those serialized bytes. And then send those encrypted bytes to the server. The server will decrypt those bytes and create the key-value pair.

 

AuthRequest

isDiffieHellmanEnabled ( NumberOfProperties PropertyId PropertyValue{ PropertyId PropertyValue}

| NumberOfEncryptedBytes EncryptedCredentials )

isDiffieHellmanEnabledfixedSize = 1 byte, type = boolean
NumberOfPropertiesfixedSize = 2 bytes, type = int16
Propertyidbytes
PropertyValuebytes
EncrptedCredentialsbytes
AuthResponse => Success
Success => fixedSize = 1 byte, type = boolean

Examples

PutRequest

string regionName = "ExampleRegion"

Key = 101

Value = "New Geode Client Server Protocol"

CallbackArg = Null

MessageHeader

 

RequestHeader

PutRequest

Size PartialMessage CorrelationIdRequestType apiVersion hasMetaDataRegionName Key CallbackArg Value ( ValueHeader value )

Size = Size of Request (65)

0x00 0x00 0x00 0x42

RequestType (PutRequestType = 3)

0x00 0x03

RegionName(type:String, value:"ExampleRegion" )

 len = 0x00 0x0d

Utf Encoding = 0x45 0x78 0x61 0x6d 0x70 0x6c 0x65 0x52 0x65 0x67 0x69 0x6f 0x6e 

Size = (number of serialized bytes = 35)

 0x00 0x00 0x00 0x23

PartialMessage = (type = Boolean, value = false)

0x00

apiVersion (1)

0x01

Key (Serialzied using geode types, value = 101)

 Geode Int type = 0x39

Value = 0x00 0x00 0x00 0x65 

isPartialBytes = (It contains all serialized bytes, type = boolean)

0x00

CorrelationId = 1

0x00 0x00 0x00 0x01 

hasMetaData (false)

0x00

CallbackArg (Serialzied using geode types, value = null)

Value = 0x29 

value (Serialized as Geode String type, value = "New Geode Client Server Protocol")

Geode String type = 0x57

Serialized Encoded length = 0x00 0x20

Encoded String = 0x4e 0x65 0x77 0x20 0x47 0x65 0x6f 0x64 0x65 0x20 0x43 0x6c 0x69 0x65 0x6e 0x74

0x20 0x53 0x65 0x72 0x76 0x65 0x72 0x20 0x50 0x72 0x6f 0x74 0x6f 0x63 0x6f 0x6c

 

PutResponse

MessageHeader

ResponseHeaderPutResponse
Size PartialMessage CorrelationIdResponseTypeId hasMetaDataSuccess

Size = Size of Request (4)

0x00 0x00 0x00 0x04

ResponseTypeId (FullResponse, type=int16, value =1)

0x00 0x01

Success(type=boolean, value = true)

0x01

PartialMessage = (type = Boolean, value = false)

0x00

hasMetaData (false)

0x00

 

CorrelationId = 1

0x00 0x00 0x00 0x01 

  

Messages

PutRequestMessagePutResponseMessage
PutRequestMessage {
	MessageHeader {
  		Size, 4 byte, int32
 		isPartialMessage, 1 byte, boolean
		CorrelationId, 4 byte, int32
 	}
 
 	RequestHeader {
 		ApiId, 2 byte, int16
 		ApiVersion, 1 byte, int8
	    hasMetaData, 1 byte, boolean
 	}
 	PutRequest {
 		regionName, varaible , String {
			len, 2 byte, int16
			variable, utf encoding		
		}
 		key, variable, bytes
		CallbaclArg, variable, bytes
 		Value {
 			ValueHeader {
				Size, 4 byte, int32
				hasPartialBytes, 1 byte, boolean
 			}
			value {
 				bytes, series of bytes
			}
 		}
 	}
}
PutResponseMessage {
	MessageHeader {
    	Size, 4 byte, int32
        isPartialMessage, 1 byte, boolean
		CorrelationId, 4 byte, int32
    }
    ResponseHeader {
 		ResponseTypeId, 2 byte, int16
 		hasMetaData, 1 byte, boolean
 	}
  	Success, 1 byte, boolean
}
GetRequestMessageGetResponseMessage
GetRequestMessage {
	MessageHeader {
  		Size, 4 byte, int32
 		isPartialMessage, 1 byte, boolean
		CorrelationId, 4 byte, int32
 	}
 
 	RequestHeader {
 		ApiId, 2 byte, int16
 		ApiVersion, 1 byte, int8
	    hasMetaData, 1 byte, boolean
 	}
 	GetRequest {
 		regionName, varaible , String {
			len, 2 byte, int16
			variable, utf encoding		
		}
 		key, variable, bytes
		CallbaclArg, variable, bytes
 	}
}
GetResponseMessage {
	MessageHeader {
    	Size, 4 byte, int32
        isPartialMessage, 1 byte, boolean
		CorrelationId, 4 byte, int32
    }
    ResponseHeader {
 		ResponseTypeId, 2 byte, int16
 		hasMetaData, 1 byte, boolean
 	}
  	Result, variable, bytes
}
PutAllRequestMessagePutAllResponseMessage
PutAllRequestMessage {
	MessageHeader {
  		Size, 4 byte, int32
 		isPartialMessage, 1 byte, boolean
		CorrelationId, 4 byte, int32
 	}
 
 	RequestHeader {
 		ApiId, 2 byte, int16
 		ApiVersion, 1 byte, int8
	    hasMetaData, 1 byte, boolean
 	}
 	PutRequest {
 		regionName, varaible , String {
			len, 2 byte, int16
			variable, utf encoding		
		}
		NumberOfKeyValuePair, 4 byte, int32 
		KeyValuePair  { 
	 		key, variable, bytes		
 			Value {
 				ValueHeader {
					Size, 4 byte, int32
					hasPartialBytes, 1 byte, boolean
 				}
				value {
 					bytes, series of bytes
				}
			}
 		}
		CallbaclArg, variable, bytes
 	}
}
PutAllResponseMessage {
	MessageHeader {
    	Size, 4 byte, int32
        isPartialMessage, 1 byte, boolean
		CorrelationId, 4 byte, int32
    }
	ResponseHeader {
 		ResponseTypeId, 2 byte, int16
 		hasMetaData, 1 byte, boolean
 	}
  	NumberOfKeysFailed , 4 byte, int32
}
GetAllRequestMessageGetAllResponseMessage
GetAllRequestMessage {
	MessageHeader {
  		Size, 4 byte, int32
 		isPartialMessage, 1 byte, boolean
		CorrelationId, 4 byte, int32
 	}
 
 	RequestHeader {
 		ApiId, 2 byte, int16
 		ApiVersion, 1 byte, int8
	    hasMetaData, 1 byte, boolean
 	}
 	GetRequest {
 		regionName, varaible , String {
			len, 2 byte, int16
			variable, utf encoding		
		}
		NumberOfKeys, 4 byte, int32 
		keys {
 			key, variable, bytes
		}
		CallbaclArg, variable, bytes
 	}
}
GetAllResponseMessage {
	MessageHeader {
    	Size, 4 byte, int32
        isPartialMessage, 1 byte, boolean
		CorrelationId, 4 byte, int32
    }
    ResponseHeader {
 		ResponseTypeId, 2 byte, int16
 		hasMetaData, 1 byte, boolean
 	}
  	NumberOfKeyValuePair, 4 byte, int32 
	KeyValuePair  { 
		key, variable, bytes		
 		value, variable, bytes
	}
}

 

RPC Frameworks

RPC frameworks such as Thrift or Apache Avro provide tools to generate client -server library based on a message schema. We are talking about how we may support various RPC frameworks to facilitate quick creation of GEODE clients in various languages supported by popular RPC frameworks.

 

  • No labels