Introduction
Geode is a reliable distributed data management tool. There is a demand to access Geode from various programming languages. But the existing client-server protocol is too complex to understand, and it’s not even documented. That establishes the need for a new client-server protocol.
Protocol Terms
Any binary protocol will require following things:
Version: This indicates the API version.
Correlation Id: This helps to relate request-response.
Object Type: What is the type of serialized object.
Response Type: It indicates whether a response is partial or complete.
ErrorCodes: It indicates the problem with API invocation.
Chunk Response: Send large response in multiple chunks.
Continuous Response: Client can register(Observer pattern) for events and then server notify the client if those events occur.
Request: It indicates client's message
Response: It indicates server's message.
Request Format: Format of request api and its parameters, which client wants to invoke.
Response Format: Format for api return value, which client invoked.
Message: Set of bytes which contain the Message Header and Request/Response.
Serialized Byte Order: Big Endian
Connect
The new protocol will be integrated with current Geode server. The new client driver can connect with Geode server by sending a protocol byte. Geode will support two protocol.
- byte - 110 : Message will contain the whole request or response.
- byte - 111 : If the message is large then client or server can divide the message into the set of small messages. Then they need to collect all the small message and parse the whole request or response.
API Type
Following table contains the api type and its corresponding Api id. The ApiId would recognize the API, which client wants to invoke on the server. The request format will contain the 1-byte(int8) for Api id. It will be marked as ApiId in the request format.
RequestType | ApiId |
---|---|
AuthenticationRequestType | 1 |
PutRequestType | 2 |
GetRequestType | 3 |
PutAllRequestType | 4 |
GetAllRequestType | 5 |
ClientConfigRequestType | 6 |
ServerConfigRequestType | 7 |
CreateRequestType | 8 |
InvalidateRequestType | 9 |
DestroyRequestType | 10 |
KeySetRequestType | 11 |
ValuesRequestType | 12 |
EntrySetRequestType | 13 |
ContainsValueForKeyRequestType | 14 |
ContainsKeyRequestType | 15 |
ContainsValueRequestType | 16 |
RemoveAllRequestType | 17 |
SizeRequestType | 18 |
PutIfAbsentRequestType | 19 |
RemoveIfValueIsSameRequestType | 20 |
ReplaceIfValueIsSameRequestType | 21 |
ReplaceIfValueExistType | 22 |
FunctionRequestType | 23 |
API Version
API version will be associated with request api. The request fromat will contain 1-byte for version. It will be marked as ApiVersion in request format. Its current value will be 1.
Correlation Id
The purpose of correlation id to match the request and its corresponding response. The request format will contain the 4-bytes(int32) for correlation Id. It will be marked as CorrelationId in request format.
Object Type
We will support all the object types which Geode understands. This would include all the primitive java types, an array of primitive types, collections, java serialization, data serializable, pdx serialization and custom user data serializers. For the purpose of the request format, we would distinguish key type and value type.
ObjectKeyType
Geode supports only few object types as the region Key. The region key will be marked as Key in the request format.
ResponseType
ReponseType will indicate that whether the response is partial or complete. A client can process a partial response. Response with FullResponse type id will indicate the completion of that request.
The response format will contain the 2-bytes(int16) for response type. It will be marked as ResponseTypeId in the response format.
ResponseType | ResponseTypeId |
---|---|
FullResponse | 1 |
PartialResponse | 2 |
Error Codes
Error codes indicate the issue with the invocation of API at the server. We have following error code for various issues at the server. The response format will contain the 2-bytes(int16) for error codes. It will be marked as ErrorCode in the response format.
Exception Type | ErrorCode |
---|---|
AUTHENICATION_REQUIRED_EXCEPTION | 1 |
AUTHORIZATION_FAILED_EXCEPTION | 2 |
AUTHETICATIONFAILED_EXCEPTION | 3 |
BUCKET_MOVED_EXCEPTION | 4 |
SERIALIZATION_EXCEPTION | 5 |
INTERRUPTED_EXCEPTION | 6 |
ILLEGAL_ARGUMNET_EXCEPTION | 7 |
ILLEGAL_STATE_EXCEPTION | 8 |
TIMEOUT_EXCEPTION | 9 |
CACHE_WRITER_EXCEPTION | 10 |
REGION_EXIST_EXCEPTION | 11 |
REGION_NOT_EXIST_EXCEPTION | 12 |
LEASE_EXPIRED_EXCEPTION | 13 |
CACHE_LOADER_EXCEPTION | 14 |
REGION_DESTROYED_EXCEPTION | 15 |
ENTRY_DESTROYED_EXCEPTION | 16 |
ENTRY_NOT_FOUND_EXCEPTION | 17 |
FUNCTION_NOT_FOUND_EXCEPTION | 18 |
FUNCTION_ATTRIBUTE_MISMATCH_EXCEPTION | 19 |
FUNCTION_EXECUTION_EXCEPTION | 20 |
CONCURRENT_MODIFICATION_EXCEPTION | 21 |
UNKNOWN_EXCEPTION | 22 |
CLASS_CAST_EXCEPTION | 23 |
GEODE_IO_EXCEPTION | 24 |
NULL_POINTER_EXCEPTION | 25 |
ENTRY_EXIST_EXCEPTION | 26 |
DISK_ACCESS_EXCEPTION | 27 |
QUERY_EXCEPTION | 28 |
CACHE_CLOSED_EXCEPTION | 29 |
MESSAGE_FORMAT_EXCEPTION | 30 |
CACHE_LISTENER_EXCEPTION | 31 |
CQ_EXCEPTION | 32 |
CQ_CLOSED_EXCEPTION | 33 |
CQ_QUERY_EXCEPTION | 34 |
CQ_EXIST_EXCEPTION | 35 |
CQ_INVALID_EXCEPTION | 36 |
INVALID_DELTA_EXCEPTION | 37 |
TRANSACTION_EXCEPTION | 38 |
TRANSACTION_DATA_NODE_DEPARTED_EXCEPTION | 39 |
TRANSACTION_REBALANCED_EXCEPTION | 40 |
COMMIT_CONFLICT_EXCEPTION | 41 |
PUTALL_PARTIAL_RESULT_EXCEPTION | 42 |
Format grammar
Format is described using Extended Backus–Naur form grammar.
ProtocolType
The format contains following fixed and variable types. They need to serialize in Big Endian byte order as showed for the example.
Type | Number Of Bytes | Value | SerializedBytes |
---|---|---|---|
boolean | Fixed = 1 | true | 0x01 |
boolean | Fixed = 1 | false | 0x00 |
int8 | Fixed = 1 | 1 | 0x01 |
int16 | Fixed = 2 | 1 | 0x00 0x01 |
int32 | Fixed = 4 | 1 | 0x00 0x00 0x00 0x01 |
int64 | Fixed = 8 | 1 | 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x01 |
String(modified UTF 8 ) | Variable
| "Geode" | 0x00 0x05 (length) 0x47 0x65 0x6f 0x64 0x65 (utf encoding) |
byte[] | Variable
| {1,2} | 0x00 0x02 (length) 0x01 0x02 |
bytes | Variable: series of bytes which contains all the meta info to create the java object. |
Message(Framing)
A message is series of bytes which contains request or response. If the message is large, then we will have provision to divide the message into small messages. The message will be sent in following way.
Message => MessageHeader (Request | Response) |
---|
MessageHeader => defined below |
Request => defined below |
Response => defined below |
MessageHeader
The Message header is a fixed size header which contains the size of a message, boolean flag to indicates whether a message is partial, and correlation id for that request message. The correlation id is used for the dual purpose here.
- If a message is sent in multiple sub-messages, then it will be used for combining the whole message.
- The Client would use correlation id to match the response to its request as well.
MessageHeader => Size isPartialMessage CorrelationId | Description |
---|---|
Size => fixedSize = 4 bytes, type = int32 | Size of request or response |
isPartialMessage => fixedSize = 1 byte, type = boolean | |
CorrelationId => fixedSize = 4 bytes, type = int32 |
Request Format
The request would contain the fixed size request header, optional metadata and request api parameters.
Request => RequestHeader [MetaData] RequestAPI |
---|
RequestHeader => defined below |
MetaData => optional |
RequestAPI => (PutRequest | GetRequest | PutAllRequest | GetAllRequest |ServerConfigRequest | ClientConfigRequest | AuthRequest) |
RequestHeader
The request header contains the ApiId, ApiVersion, and hasMetaData flag to indicate whether the request contains some metadata.
RequestHeader => ApiId apiVersion hasMetaData | Description |
---|---|
ApiId => fixedSize = 2 bytes, type = int16 | |
apiVersion => fixedSize = 1 byte, type = int8 | |
hasMetaData => fixedSize = 1 byte, type = boolean | if there is any meta data associated with this request |
Response Format
The response would contain the fixed size response header, optional metadata and return values.
Response => ResponseHeader [MetaData] APIResponse |
---|
ResponseHeader => defined below |
MetaData => Optional |
APIResponse => (PutResponse | GetResponse | PutAllResponse | GetAllResponse | ServerConfigResposne | ClientConfigResponse | AuthResponse) |
ResponseHeader
The response header will have resposneType, which indicates its partial response, full response or error. A hasMetaData flag indicates whether the response contains some metadata.
ResponseHeader => (ResponseTypeId | ErrorCode) hasMetaData CorrelationId | Description |
---|---|
ResponseTypeId => fixedSize = 2 bytes, type = int16 | |
ErrorCode => fixedSize = 2 bytes, type = int16 | |
hasMetaData => fixedSize = 1 byte, type = boolean | if there is any meta data associated with this request |
CorrelationId => fixedSize = 4 bytes, type = int32 |
Value
The Value is serialized bytes for the Geode region value. It contains value header and series of bytes. Using value header, we can send a large number of bytes in more than one chunk.
Value => ValueHeader value {ValueHeader value} | Description |
---|---|
ValueHeader => defined below | |
value => bytes | Serialized Value Object which Geode can understand |
ValueHeader
The value header contains the value bytes size, and a flag indicates whether it contains all the value bytes.
ValueHeader => Size hasPartialBytes | Description |
---|---|
Size => fixedSize = 4 bytes, type = int32 | Number of serialized bytes |
hasPartialBytes => fixedSize = 1 byte, type = boolean | Whether this contains partial bytes of value |
APIS
PutRequest
PutRequest => RegionName Key CallbackArg Value |
---|
RegionName => (variable size, type = String) |
Key => bytes |
CallbackArg => bytes |
Value => defined above |
PutResponse
PutResponse => Success |
---|
Success => fixedSize = 1 byte, type = boolean |
GetRequest
GetRequest --> RegionName Key CallbackArg |
---|
RegionName => (variable size, type = String) |
Key => bytes |
CallbackArg => bytes |
GetResponse
GetResponse => Result |
---|
Result => bytes |
GetAllRequest
GetAllRequest => RegionName NumberOfKeys Key {Key} CallbackArg |
---|
RegionName => (variable size, type = String) |
NumberOfKeys => fixedSize = 4 bytes, type = int32 |
Key => bytes |
CallbackArg => bytes |
GetAllResponse
GetAllResponse --> NumberOfKeyValuePair Key value {Key value} |
---|
NumberOfKeyValuePair => fixedSize = 4 bytes, type = int32 |
Key => bytes |
value => bytes |
PutAllRequest
PutAllRequest => RegionName NumberOfKeys Key Value{Key Value} CallbackArg |
---|
RegionName => (variable size, type = String) |
NumberOfKeys => fixedSize = 4 bytes, type = int32 |
Key => bytes |
Value => defined above |
CallbackArg => bytes |
PutAllResponse
PutAllResponse => Success |
---|
Success => fixedSize = 1 byte, type = boolean |
Metadata
The purpose of metadata to pass defined key value pair with request and response. That will be optional for client. If there is any metadata associated with request or response, then need to set "hasMetadata" flag to "true" in request or response header. After that send metadata in following format.
MetaData => NumberOfMetadata MatadataKeyId MetadataKeyValue { MatadataKeyId MetadataKeyValue} | |
---|---|
NumberOfMetadata=> fixedSize = 2 bytes, type = int16 | |
MatadataKeyId => fixedSize = 2 bytes, type = int16 | |
MetadataKeyValue => variable, Value as defined in table below |
We would have following pre-defined key and value for metadata. Note this list will grow over time.
Request MetaData Key | MetaData KeyId | MetaData Value | Description |
---|---|---|---|
JSON_KEY | 1 fixedSize = 2 bytes, type = int16 | true fixedSize = 1 byte, type = boolean | Geode will expect key as JSON string(or bytes) and convert that into pdx key. If the response requires a key, then it will convert pdx key to JSON string(or bytes) back. |
JSON_VALUE | 2 fixedSize = 2 bytes, type = int16 | true fixedSize = 1 byte, type = boolean | Geode will expect Value as JSON string(or bytes) and convert that into pdx value. If the response requires a value, then it will convert pdx value to JSON string(or bytes) back. |
EVENT_ID | 3 fixedSize = 2 bytes, type = int16 | EventId { uniqueId: type = String ThreadId:type=int64 SequenceId: type=int64 } | The eventid is used to identify same region event in Geode. Geode keeps map of "uniqueId + threadId" Vs "SequenceId" to know whether region event has been already seen or not.
|
Response MetaData Key | MetaData KeyId | MetaData Value | Description |
---|---|---|---|
UPDATE_PR_META_DATA | 1 fixedSize = 2 bytes, type = int16 | true fixedSize = 1 byte, type = boolean | [optional]This would indicate in response that request was handled by remote peer. So client should update PR meta data. |
ServerConfigRequest
The purpose of ServerConfigRequest request to get server config from the server. The client needs to send this request after connecting to the server. If the client knows server properties, then there is no need to send this request.
ServerConfigRequest => {} |
---|
ServerConfigResponse =>NumberOfProperties PropertyId PropertyValue{ PropertyId PropertyValue} | |
---|---|
NumberOfProperties => fixedSize = 2 bytes, type = int16 | |
Propertyid => fixedSize = 2 bytes, type = int16 | |
PropertyValue => variable, Value as defined in table below |
Server Response Properties | PropertyId | PropertyValue | Description |
---|---|---|---|
SECURITY_ENABLED | 1 | boolean | Whether security is enabled at server |
DIFFIE_HELLMAN_KEY | 2 | byte[] | The server Diffie-Hellman key if the credential is required to encrypt. |
MAX_TIME_BETWEEN_CLIENT_PING | 3 | int32 | If the client connection is idle for MAX_TIME_BETWEEN_CLIENT_PING seconds then the server will close that connection. |
ClientConfigRequest
The Purpose of ClientConfigRequest request to send client config to a server. The client needs to send this request after connecting to the server.
ClientConfigRequest | NumberOfProperties PropertyId PropertyValue{ PropertyId PropertyValue} |
---|---|
NumberOfProperties | fixedSize = 2 bytes, type = int16 |
Propertyid | fixedSize = 2 bytes, type = int16 |
PropertyValue | variable, Value as defined in table below |
ClientConfigResponse –-> Success |
---|
Success => fixedSize = 1 byte, type = boolean |
Client Request Properties | PropertyId | PropertyValue | Description | |
---|---|---|---|---|
CLIENT_ID | 1 | string | Unique id for the client; if client doesn't send this property then server will just create id (client-host, client-port, server-host, server-port). | |
ClIENT_READ_TIMEOUT | 2 | int32 | The client will wait for ClIENT_READ_TIMEOUT for server response. It's a optional property. | |
DIFFIE_HELLMAN_KEY | 3 | byte[] | The client Diffie-Hellman key if the credential is required to encrypt. If server has enabled Diffie-Hellman then client must send this to server. |
AuthRequest
The purpose of AuthRequest to authenticate the client connection. A client can send the auth request in the following format. Before sending the auth request a client can verify the server config by sending the ServerConfigRequest. A client can send key-value pair of credentials to authenticate itself.
Those pair needs to serialized in following way. If Diffie-Hellman encryption is enabled on the server side then need to encrypt those serialized bytes. And then send those encrypted bytes to the server. The server will decrypt those bytes and create the key-value pair.
AuthRequest | ( NumberOfProperties PropertyId PropertyValue{ PropertyId PropertyValue} | NumberOfEncryptedBytes EncryptedCredentials ) |
---|---|
NumberOfProperties | fixedSize = 2 bytes, type = int16 |
Propertyid | bytes |
PropertyValue | bytes |
EncrptedCredentials | bytes |
AuthResponse –-> Success |
---|
Success => fixedSize = 1 byte, type = boolean |
Examples
PutRequest
string regionName = "ExampleRegion"
Key = 101
Value = "New Geode Client Server Protocol"
CallbackArg = Null
MessageHeader
| RequestHeader | PutRequest | |
---|---|---|---|
Size PartialMessage CorrelationId | RequestType apiVersion hasMetaData | RegionName Key CallbackArg | Value ( ValueHeader value ) |
Size = Size of Request (65) 0x00 0x00 0x00 0x42 | RequestType (PutRequestType = 3) 0x00 0x03 | RegionName(type:String, value:"ExampleRegion" ) len = 0x00 0x0d Utf Encoding = 0x45 0x78 0x61 0x6d 0x70 0x6c 0x65 0x52 0x65 0x67 0x69 0x6f 0x6e | Size = (number of serialized bytes = 35) 0x00 0x00 0x00 0x23 |
PartialMessage = (type = Boolean, value = false) 0x00 | apiVersion (1) 0x01 | Key (Serialzied using geode types, value = 101) Geode Int type = 0x39 Value = 0x00 0x00 0x00 0x65 | isPartialBytes = (It contains all serialized bytes, type = boolean) 0x00 |
CorrelationId = 1 0x00 0x00 0x00 0x01 | hasMetaData (false) 0x00 | CallbackArg (Serialzied using geode types, value = null) Value = 0x29 | value (Serialized as Geode String type, value = "New Geode Client Server Protocol") Geode String type = 0x57 Serialized Encoded length = 0x00 0x20 Encoded String = 0x4e 0x65 0x77 0x20 0x47 0x65 0x6f 0x64 0x65 0x20 0x43 0x6c 0x69 0x65 0x6e 0x74 0x20 0x53 0x65 0x72 0x76 0x65 0x72 0x20 0x50 0x72 0x6f 0x74 0x6f 0x63 0x6f 0x6c |
PutResponse
MessageHeader | ResponseHeader | PutResponse |
---|---|---|
Size PartialMessage CorrelationId | ResponseTypeId hasMetaData | Success |
Size = Size of Request (4) 0x00 0x00 0x00 0x04 | ResponseTypeId (FullResponse, type=int16, value =1) 0x00 0x01 | Success(type=boolean, value = true) 0x01 |
PartialMessage = (type = Boolean, value = false) 0x00 | hasMetaData (false) 0x00 | |
CorrelationId = 1 0x00 0x00 0x00 0x01 |
Messages
PutRequestMessage | PutResponseMessage |
---|---|
PutRequestMessage { MessageHeader { Size, 4 byte, int32 isPartialMessage, 1 byte, boolean CorrelationId, 4 byte, int32 } RequestHeader { ApiId, 2 byte, int16 ApiVersion, 1 byte, int8 } PutRequest { regionName, varaible , String { len, 2 byte, int16 variable, utf encoding } key, variable, bytes CallbaclArg, variable, bytes Value { ValueHeader { Size, 4 byte, int32 hasPartialBytes, 1 byte, boolean } value { bytes, series of bytes } } } } | PutResponseMessage { MessageHeader { Size, 4 byte, int32 isPartialMessage, 1 byte, boolean CorrelationId, 4 byte, int32 } Success, 1 byte, boolean } |
GetRequestMessage | GetResponseMessage |
GetRequestMessage { MessageHeader { Size, 4 byte, int32 isPartialMessage, 1 byte, boolean CorrelationId, 4 byte, int32 } RequestHeader { ApiId, 2 byte, int16 ApiVersion, 1 byte, int8 } GetRequest { regionName, varaible , String { len, 2 byte, int16 variable, utf encoding } key, variable, bytes CallbaclArg, variable, bytes } } | GetResponseMessage { MessageHeader { Size, 4 byte, int32 isPartialMessage, 1 byte, boolean CorrelationId, 4 byte, int32 } Result, variable, bytes } |
PutAllRequestMessage | PutAllResponseMessage |
PutAllRequestMessage { MessageHeader { Size, 4 byte, int32 isPartialMessage, 1 byte, boolean CorrelationId, 4 byte, int32 } RequestHeader { ApiId, 2 byte, int16 ApiVersion, 1 byte, int8 } PutRequest { regionName, varaible , String { len, 2 byte, int16 variable, utf encoding } NumberOfKeyValuePair, 4 byte, int32 KeyValuePair { key, variable, bytes Value { ValueHeader { Size, 4 byte, int32 hasPartialBytes, 1 byte, boolean } value { bytes, series of bytes } } } CallbaclArg, variable, bytes } } | PutAllResponseMessage { MessageHeader { Size, 4 byte, int32 isPartialMessage, 1 byte, boolean CorrelationId, 4 byte, int32 } Success, 1 byte, boolean } |
GetAllRequestMessage | GetAllResponseMessage |
GetAllRequestMessage { MessageHeader { Size, 4 byte, int32 isPartialMessage, 1 byte, boolean CorrelationId, 4 byte, int32 } RequestHeader { ApiId, 2 byte, int16 ApiVersion, 1 byte, int8 } GetRequest { regionName, varaible , String { len, 2 byte, int16 variable, utf encoding } NumberOfKeys, 4 byte, int32 keys { key, variable, bytes } CallbaclArg, variable, bytes } } | GetAllResponseMessage { MessageHeader { Size, 4 byte, int32 isPartialMessage, 1 byte, boolean CorrelationId, 4 byte, int32 } NumberOfKeyValuePair, 4 byte, int32 KeyValuePair { key, variable, bytes Value { ValueHeader { Size, 4 byte, int32 hasPartialBytes, 1 byte, boolean } value { bytes, series of bytes } } } } |