...
ID | IEP-76 |
Author | |
Sponsor | |
Created | |
Status | |
Motivation
...
Adapt Ignite 2.x protocol for Ignite 3.0. Main The main differences are:
- MsgPack as underlying binary format (IEP-75 Thin Client MsgPack Serialization)
- Different A different set of data types (IEP-54: Schema-first Approach)
- More flexible and extensible handshake with magic header string
...
- Every Ignite node listens on a TCP port. Thin client implementations connect to any node in the cluster (possibly multiple nodes) through a TCP socket and perform Ignite operations using a well-defined binary protocol.
- Server-side connection parameters are defined in
ClientConnectorConfiguration
class.- Default port is 10800.
- Connector is enabled by default, no configuration changes are needed.
- Netty is used on the server for network IO.
...
MsgPack is used for data serialization (uses big-endian byte order).
Data Types Mapping
Ignite data types defined in IEP-54 map to MsgPack data types the following way:
Ignite Type | Size | MsgPack Type | Notes |
---|
Bitmask(n) | ⌈n/8⌉ bytes | bin8 / bin16 / bin32 | Extension 8 |
IntX, UintX | 1-8 bytes | fixint / intX / uintX | Integer types are interchangeable when possible. fixint (1 byte) can be passed as a value for uint64 column. |
Float | 4 bytes | float32 |
|
Double | 8 bytes | float64 |
|
Number([n]) | Variable | ext16 (up to 2^8 - 1 bytes) | Extension 1 |
Decimal | Variable | ext16 (up to 2^8 - 1 bytes) | Extension 2 |
UUID | 16 bytes | fixext16 | Extension 3 |
String | Variable | str |
|
Date | 3 bytes | fixext4 | Extension 4 |
Time | 4 5 bytes | fixext4ext8 | Extension 5 |
Datetime | 7 bytes | fixext8 | Extension 6 |
Timestamp | 8 bytes | 10 bytes | ext8 | Extension 7 |
IgniteUuid | 24 bytes | ext8 | Extension 9 |
NoValue | 1 byte | fixext1 | Extension 10timestamp 64 |
Binary | Variable | bin32 (up to 2^32 - 1 bytes) |
|
"Extension X" is a MsgPack extension type (up to 128 extension types are allowed).
Integer data types, varint encoding
MsgPack provides multiple data types for integer values. When encoding a value, the smallest data type for that value is picked automatically.
"int" is used below to denote int format family. Values such as payload size, request ID, error codes, are encoded this way - using variable number of bytes based on the value.
Message format
All messages,
...
requests and
...
responses, except handshake, start with
...
a 4-byte message length (excluding the length itself).
...
- Message length is not encoded with MsgPack, 4 bytes contain raw int32 value in little-endian byte order.
- Maximum message length is 2147483647 (Integer.MAX_VALUE and maximum byte array length in Java)
4 bytes | Length of Payload |
... | Payload |
Tuple serialization
- This IEP covers Table API - Ignite#tables(), which is the only public API available at the moment.
- Table API operates on tuples (Tuple and TupleBuilder interfaces).
- Tuple is a set of key-value pairs - column name and value.
- Values can be of basic types described above and in IEP-54, according to the table schema.
- Table schema can evolve over time, columns get added and removed. Ignite tracks the evolution with incrementing integer schema version.
- All schema versions are stored by Ignite. It is possible to retrieve the set of columns for any version.
A simple way to serialize a tuple would be to write a map (String → ...) from column name to value.
However, thanks to schema-first approach, we can avoid sending column names with the values (serializing strings is expensive). Instead, we can write an integer schema version, and then values for every column in that schema.
values "type" below indicates a raw sequence of values (not a MsgPack array). Since we know the number of columns in the schema, we don't need an array header.
int | Schema version |
values | Column values in schema order. nil when there is no value for a particular column. |
To read or write a value for a column, we get the column type from the schema and use corresponding MsgPack data type according to Data Types Mapping above.
Code Block |
---|
language | java |
---|
title | Tuple serialization (pseudocode) |
---|
|
for (Column col : tuple) {
switch (col.type().spec()) {
case BYTE:
packer.packByte(tuple.byteValue(col.name()));
break;
case SHORT:
... |
Key tuples
Key tuples are tuples with key columns only. Key columns always come first in the schema. So if there are 2 key columns, first two values of the tuple is the key.
Null vs NoValue
Ignite Table API handles "column set to null" (1) and "column not set" (2) differently.
- Non-nullable column does not allow (1), but allows (2) as long as there is a default value.
- Nullable column will be set to null in (1) and to default value in (2).
NoValue custom protocol type reflects this distinction.
Handshake
Request |
---|
4 bytes | Magic number 49 47 4E 49, or "IGNI". Helps to identify misconfigured SSL or other apps probing the port. |
int | Payload length |
int | Version major |
int | Version minor |
int | Version patch |
int | Client code (1 for JDBC, 2 for general-purpose client) |
bin | Features (bitmask) |
map (string → any) | Extensions (auth, attributes, etc). Server can skip unknown extension data (when client is newer). |
Response |
---|
4 bytes | Magic number 49 47 4E 49, or "IGNI". Helps to identify misconfigured SSL or different server on the port. |
int | Payload length |
int | Server version major |
int | Server version minor |
int | Server version patch |
int | Error code (0 for success) |
string | Error message (when error code is not 0) |
int | When error code is 0: Server idle timeout |
string | When error code is 0: Server node id |
string | When error code is 0: Server node name (consistent id) |
bin | When error code is 0: Server features (bitmask) |
map (string → any) | When error code is 0: Extensions (auth, attributes, etc). Client can skip unknown extension data (when server is newer). |
Client Operations
Upon successful handshake, client can start performing operations by sending a request with specific op code. Each operation has its own request and response format, with a common header.
Request |
int | Operation code |
int | Request id, generated by client and returned as-is in response |
... | Operation-specific data |
Response |
int | Type = 0 |
int | Request id |
int | Flags (1 = partition assignment changed) |
long | Observable timestamp (causality token) |
uuid or null | Trace id (null for success) |
int | Error code (when trace id is not null) |
string | Error message (when trace id is not null) |
string or null | Error stack trace (when trace id is not null) |
map or null | Error details (when trace id is not null) |
... | Operation-specific data (when trace id is null) |
Request or response without operation-specific data is called basic request or basic response below.
Notifications
Clients should expect notification messages at any moment after the successful handshake.
Notification |
int | Type = 1 |
int | Notification code |
... | Notification-specific data |
Operation codes and request ids are omitted everywhere below for brevity.
TABLES_GET = 3
Basic request.
Response |
---|
map (UUID -> string) | map of table ids and names |
Note: tables can only be created/deleted with SQL, there are no TABLE_CREATE or TABLE_DROP operations.
TABLE_GET = 4
Response |
---|
UUID or nil | table ID or null when a table with the given name does not exist |
SCHEMAS_GET = 5
Request |
---|
UUID | table ID |
arr or nil | schema IDs, or null to get latest |
Response |
---|
map (int → array (array)) | Map from schema ID to columns. Column is represented by an array of values for: name, type, isKey, isNullable. The array can contain extra data in future for additional properties. |
TUPLE_UPSERT = 10
Request |
---|
UUID | table ID |
int or nil | transaction ID |
int | schema ID |
values | values for all columns in given schema (nil when value is missing for a column) |
Basic response.
Client side is supposed to match provided columns against the latest known schema.
- If columns don't match, request the latest schema and try again.
- If the latest schema still does not match, and live schema is enabled, use TUPLE_UPSERT_SCHEMALESS
TUPLE_GET = 12
Request |
---|
UUID | table ID |
int or nil | transaction ID |
int | schema ID |
values | values for key columns (in schema order) |
Response |
---|
int or nil | schema id for the current tuple, or nil when there is no matching record |
values | values for value columns in schema order, when schema id is not nil |
Clients should retrieve schemas with SCHEMAS_GET and cache them per table.
TUPLE_UPSERT_ALL = 13
Request |
---|
UUID | table ID |
int or nil | transaction ID |
int | schema ID |
int | row count |
values | rows with values for all columns in given schema (nil when value is missing for a column) |
Basic response.
TUPLE_GET_ALL = 15
Request |
---|
UUID | table ID |
int or nil | transaction ID |
int | schema ID, or nil when result set is empty |
int | row count |
values | array of rows with values for key columns (in schema order) |
Response |
---|
int | schema ID (for all tuples in response) |
int | row count |
values | rows with values in schema order |
TUPLE_GET_AND_UPSERT = 16
Request |
---|
UUID | table ID |
int or nil | transaction ID |
int | schema ID |
values | values for all columns in given schema (nil when value is missing for a column) |
Response |
---|
int or nil | schema id for the current tuple, or nil when there is no matching record |
values | values for value columns in schema order |
TUPLE_INSERT = 18
Request |
---|
UUID | table ID |
int or nil | transaction ID |
int | schema ID |
values | values for all columns in given schema (nil when value is missing for a column) |
Response |
---|
bool | Insert result |
TUPLE_INSERT_ALL = 20
Request |
---|
UUID | table ID |
int or nil | transaction ID |
int | schema ID |
int | row count |
values | rows with values for all columns in given schema (nil when value is missing for a column) |
Response |
---|
int | schema id, or nil when no rows were skipped |
int | skipped row count |
values | skipped rows (values in schema order) |
TUPLE_REPLACE = 22
Request |
---|
UUID | table ID |
int or nil | transaction ID |
int | schema ID |
values | values for all columns in given schema (nil when value is missing for a column) |
Response |
---|
bool | Replace result |
TUPLE_REPLACE_EXACT = 24
Request |
---|
UUID | table ID |
int or nil | transaction ID |
int | schema ID |
values | oldRec: values for all columns in given schema (nil when value is missing for a column) |
values | newRec: values for all columns in given schema (nil when value is missing for a column) |
Response |
---|
bool | Replace result |
TUPLE_GET_AND_REPLACE = 26
Request |
---|
UUID | table ID |
int or nil | transaction ID |
int | schema ID |
values | values for all columns in given schema (nil when value is missing for a column) |
Response |
---|
int or nil | schema id for the current tuple, or nil when there is no matching record |
values | values for value columns in schema order |
TUPLE_DELETE = 28
Request |
---|
UUID | table ID |
int or nil | transaction ID |
int | schema ID |
values | values for key columns in given schema |
Response |
---|
bool | Delete result |
TUPLE_DELETE_ALL = 29
Request |
---|
UUID | table ID |
int or nil | transaction ID |
int | schema ID |
int | row count |
values | rows with values for key columns in a given schema |
Response |
---|
int | schema id, or nil when no rows were skipped |
int | skipped row count |
values | skipped rows (values for key columns in schema order) |
TUPLE_DELETE_EXACT = 30
Request |
---|
UUID | table ID |
int or nil | transaction ID |
int | schema ID |
values | values for all columns in given schema (nil when value is missing for a column) |
Response |
---|
bool | Delete result |
TUPLE_DELETE_ALL_EXACT = 31
Request |
---|
UUID | table ID |
int or nil | transaction ID |
int | schema ID |
int | row count |
values | rows with values for all columns in given schema (nil when value is missing for a column) |
Response |
---|
int | schema id, or nil when no rows were skipped |
int | skipped row count |
values | skipped rows (values for key columns in schema order) |
TUPLE_GET_AND_DELETE = 32
Request |
---|
UUID | table ID |
int or nil | transaction ID |
int | schema ID |
values | values for all columns in given schema (nil when value is missing for a column) |
Response |
---|
int or nil | schema id for the current tuple, or nil when there is no matching record |
values | values for value columns in schema order, when schema id is not null |
TUPLE_CONTAINS_KEY = 33
Request |
---|
UUID | table ID |
int or nil | transaction ID |
int | schema ID |
values | values for key columns |
Response |
---|
bool | whether a tuple with the given key exists |
TX_BEGIN = 43
Request |
---|
bool | Read-only tx flag |
long | Observable timestamp (value from server - see standard response header) |
Response |
---|
int | Transaction ID |
TX_COMMIT = 44
Basic response
TX_ROLLBACK = 45
Basic response
Risks and Assumptions
- This IEP covers handshake and Tables API, which is the only public API available at the moment.
- Invoke API is out of scope: code deployment and processor serialization should be designed separately.
Discussion Links
Reference Links
...
Handshake
TODO: Magic word, features as map ![(question)](/confluence/s/34gb19/8804/10mvnxf/_/images/icons/emoticons/help_16.svg)
Risks and Assumptions
TODO
Discussion Links
...
...