Motivation

Thin client protocol will be the primary way to interact with Ignite 3.0 from code.

Description

Adapt Ignite 2.x protocol for Ignite 3.0. Main differences are:

• MsgPack as underlying binary format (IEP-75 Thin Client MsgPack Serialization)
• Different set of data types (IEP-54: Schema-first Approach)
• More flexible and extensible handshake with magic header string

TCP Socket

• Every Ignite node listens on a TCP port. Thin client implementations connect to any node in the cluster (possibly multiple nodes) through a TCP socket and perform Ignite operations using a well-defined binary protocol.
• Server-side connection parameters are defined in ClientConnectorConfiguration class.
  • Default port is 10800.
  • Connector is enabled by default, no configuration changes needed.
• Netty is used on the server for network IO.

Data Format

MsgPack is used for data serialization.

Data Types Mapping

Ignite data types defined in IEP-54 map to MsgPack data types the following way:

"Extension X" is a MsgPack extension type (up to 128 extension types are allowed).

Integer data types, varint encoding

MsgPack provides multiple data types for integer values. When encoding a value, smallest data type for that value is picked automatically.

"int" is used below to denote int format family. Values such as payload size, request ID, error codes, are encoded this way - using variable number of bytes based on the value.

Message format

All messages, request and response, except handshake, start with int message length (excluding the length itself).

• Any MsgPack int type can be used (see varint above). Empty message will be single 0 byte.
• Maximum message length is 2147483647 (Integer.MAX_VALUE and maximum byte array length in Java)

Tuple serialization

• This IEP covers Table API - Ignite#tables(), which is the only public API available at the moment.
• Table API operates on tuples (Tuple and TupleBuilder interfaces).
• Tuple is a set of key-value pairs - column name and value.
• Values can be of basic types described above and in IEP-54, according to the table schema. 
• Table schema can evolve over time, columns get added and removed. Ignite tracks the evolution with incrementing integer schema version.
• All schema versions are stored by Ignite. It is possible to retrieve the set of columns for any version.

A simple way to serialize a tuple would be to write a map (String → ...) from column name to value.

However, thanks to schema-first approach, we can avoid sending column names with the values (serializing strings is expensive). Instead, we can write integer schema version, and then an array of values for every column in that schema.

To read or write a value for a column, we get the column type from the schema and use corresponding MsgPack data type according to Data Types Mapping above.

for (Column col : tuple) {
	switch (col.type().spec()) {
    	case BYTE:
        	packer.packByte(tuple.byteValue(col.name()));
	        break;
	    case SHORT:
...

Key tuples

Key tuples are tuples with key columns only. Key columns always come first in the schema. So if there are 2 key columns, first two values of the tuple is the key.

Dynamic schema expansion (Live-schema)

IEP-54: Schema-first Approach includes "dynamic schema expansion" feature, which means that users can insert arbitrary tuples and Ignite will automatically update the schema with extra fields, if any.

To support this feature, we provide schemaless alternatives for all write operations (upsert, insert, etc).

Handshake

Client Operations

 Upon successful handshake client can start performing operations by sending a request with specific op code. Each operation has it's own request and response format, with a common header.

Request or response without operation-specific data is called basic request or basic response below.

Notifications

Clients should expect notification messages at any moment after the successful handshake.

Operation codes and request ids are omitted everywhere below for brevity.

TABLE_CREATE = 1

TABLE_DROP = 2

Basic response.

TABLES_GET = 3

Basic request.

TABLE_GET = 4

SCHEMAS_GET = 5

TUPLE_UPSERT = 10

Basic response.

Client side is supposed to match provided columns against the latest known schema. If columns don't match, request the latest schema and try again.

TUPLE_GET = 11

Clients should retrieve schemas with SCHEMAS_GET and cache them per table.

TUPLE_UPSERT_ALL = 12

Basic response.

TUPLE_GET_ALL = 13

TUPLE_GET_AND_UPSERT = 14

TUPLE_INSERT = 15

Basic response.

TUPLE_INSERT_ALL = 16

Basic response.

TUPLE_REPLACE = 17

TUPLE_REPLACE2 = 18

TUPLE_GET_AND_REPLACE = 19

TUPLE_DELETE = 20

TUPLE_DELETE_ALL = 21

TUPLE_DELETE_EXACT = 22

TUPLE_DELETE_ALL_EXACT = 23

TUPLE_GET_AND_DELETE = 24

Risks and Assumptions

• This IEP covers handshake and Tables API, which is the only public API available at the moment.
• Invoke API is out of scope: code deployment and processor serialization should be designed separately.

Discussion Links

• https://lists.apache.org/thread.html/r719c076b9d2423c99f471075c0c3bae7642d693331bc1767b4d87e32%40%3Cdev.ignite.apache.org%3E

Reference Links

• PoC: https://github.com/apache/ignite-3/pull/191
• IEP-75 Thin Client MsgPack Serialization
• IEP-9 Thin Client Protocol (Ignite 2.x protocol)

Tickets