...
ID | IEP-75 | ||||||||||
Author | |||||||||||
Sponsor | |||||||||||
Created |
| ||||||||||
Status |
|
Table of Contents |
---|
Thin clients need a standardized way to serialize data for Table and Key-Value APIs.
In 2.x , we used the same Ignite binary format that was used for server communication and data storage. However, in 3.x there are different formats for data storage and transmission (TODO: linksIEP-74 Data Storage), and those formats are not meant to be used by thin clients.
In Ignite 3.0 we want to avoid the cost of writing and supporting our own serialization mechanism.
Thin client protocol (handshake, message format, etc) will be designed in a separate IEP. Here we only discuss a mechanism to serialize user and system data: primitive and compound values, such as cache entries, configuration objects, and so on.
...
Use MsgPack format in the Ignite 3.0 thin client protocol.
Code Block | ||||
---|---|---|---|---|
| ||||
packer
.packString("select * from cars where year > ? and seats = ?")
.packArrayHeader(2)
.packInt(2005)
.packInt(2); |
Result is 54 bytes:
For comparison, current Ignite binary protocol encodes the same data in 71 bytes, and takes 2x time to do so (see benchmarks below - writeSqlQueryIgnite and writeSqlQueryMsgPack).
The goal is to find an existing serialization format that satisfies the following requirements:
Name | Comments | License | |||||
---|---|---|---|---|---|---|---|
MessagePack |
| Java: Apache 2.0, C#: MIT, C++: MIT (nlohmann/json), TODOPython: Apache 2.0, JavaScript: MIT, PHP: MIT | |||||
CBOR | CBOR |
|
| Protobuf | Avro | Thrift | BSON |
// TODO: ?
// TODO
Java: Apache 2.0, C#: CC0, C++: MIT, Python: MIT, JavaScript: MIT, PHP: PHP License | ||
FlexBuffers |
| |
BSON | Designed for MongoDB storage and in-memory manipupation, not for network usage => more verbose than MessagePack/CBOR | |
UBJSON | Seems to be abandoned, implementations (e.g. C#) are not maintained |
Popular formats like Avro, Thrift, ProtoBuf, FlatBuffers and others are not mentioned, because the don't satisfy one or more requirements above (schemaless, etc).
MessagePack is more widely used and has more mature and well-maintained implementations in all languages of interest.
No Format |
---|
Benchmark Mode Cnt Score Error Units
JmhBinaryMarshallerMsgPackBenchmark.writePrimitivesMsgPackRaw thrpt 10 16834154.556 ± 85624.143 ops/s
JmhBinaryMarshallerMsgPackBenchmark.writePrimitivesIgnite thrpt 10 12702562.838 ± 248094.068 ops/s
JmhBinaryMarshallerMsgPackBenchmark.writePojoIgnite thrpt 10 11590924.790 ± 42061.734 ops/s // Full footers
JmhBinaryMarshallerMsgPackBenchmark.writePojoMsgPack thrpt 10 5386377.535 ± 33835.097 ops/s // Fields with names
JmhBinaryMarshallerMsgPackBenchmark.writePojoMsgPack2 thrpt 10 8505961.494 ± 465369.449 ops/s // Fields without names
JmhBinaryMarshallerMsgPackBenchmark.readPrimitivesIgnite thrpt 10 19873521.096 ± 545779.558 ops/s
JmhBinaryMarshallerMsgPackBenchmark.readPrimitivesMsgPack thrpt 10 29235107.372 ± 85371.004 ops/s
JmhBinaryMarshallerMsgPackBenchmark.readPojoIgnite thrpt 10 8437054.066 ± 104476.415 ops/s
JmhBinaryMarshallerMsgPackBenchmark.readPojoMsgPack thrpt 10 6292876.474 ± 73356.915 ops/s
JmhBinaryMarshallerMsgPackBenchmark.writeSqlQueryIgnite thrpt 10 5756908.336 ± 42079.083 ops/s
JmhBinaryMarshallerMsgPackBenchmark.writeSqlQueryMsgPack thrpt 10 12380076.956 ± 150712.634 ops/s
(Ubuntu 20.04, OpenJDK 1.8.0_292, i7-9700K) |
Jira | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|
|