You are viewing an old version of this page. View the current version.
Compare with Current
View Page History
« Previous
Version 37
Next »
ID | IEP-75 |
Author | |
Sponsor | |
Created | |
Status | DRAFT |
Motivation
Thin clients need a standardized way to serialize data for Table and Key-Value APIs. In 2.x, we used the same Ignite binary format that was used for server communication and data storage. However, in 3.x there are different formats for data storage and transmission (IEP-74 Data Storage), and those formats are not meant to be used by thin clients.
In Ignite 3.0 we want to avoid the cost of writing and supporting our own serialization mechanism.
Non-goals
Thin client protocol (handshake, message format, etc) will be designed in a separate IEP. Here we only discuss a mechanism to serialize user and system data: primitive and compound values, such as cache entries, configuration objects, and so on.
Description
Requirements
The goal is to find an existing serialization format that satisfies the following requirements:
- Binary (as opposed to text, like JSON, for performance reasons)
- Supports nested object graphs
- Supports primitives, not only objects (for example, integer or Guid value can be serialized independently)
- Supports streaming: multiple values one after another in the same buffer / stream.
- Schemaless: any object of any type can be written without prior set up
- For Table APIs (when the schema is present), we can use field IDs instead of names for performance reasons
- Can work without classes ("binary mode" in terms of 2.x): servers should be able to inspect the structure in serialized form
- Extensible (can add custom types)
- Well-supported implementations in all languages of interest (Java, C#, C++, Python, JavaScript, PHP)
- Fast and compact
Comparison
Name | Comments | License |
---|
MessagePack | - Schemaless binary format.
- Compatible with JSON (can be directly converted convert to JSON and from JSON: an important use case)
- The most popular among all. High-performance, well-maintained implementations exist for many languages.
- Battle tested: used by Redis
| Java: Apache 2.0, C#: MIT, C++: MIT (nlohmann/json), Python: Apache 2.0, JavaScript: MIT, PHP: MIT |
CBOR | - Based on MessagePack.
- Less popular than MessagePack, fewer implementations, outdated PHP implementation.
- Standardized (RFC7049), but MessagePack is simpler.
- Included in stdlib in .NET 5.
- "Use MsgPack instead of CBOR": https://diziet.dreamwidth.org/6568.html
| Java: Apache 2.0, C#: CC0, C++: MIT, Python: MIT, JavaScript: MIT, PHP: PHP License |
FlexBuffers | - "Schemaless cousin of Google's FlatBuffers". Can be accessed without parsing, copying, or allocation.
- Can't serialize arbitrary objects at this point (in Java and C#)
- Relatively new, has not gained traction
|
|
BSON | Designed for MongoDB storage and in-memory manipupation, not for network usage => more verbose than MessagePack/CBOR |
|
UBJSON | Seems to be abandoned, implementations (e.g. C#) are not maintained |
|
Popular formats like Avro, Thrift, ProtoBuf, FlatBuffers and others are not mentioned, because the don't satisfy one or more requirements above (schemaless, etc).
Conclusion
- MessagePack and CBOR satisfy all requirements (and they are very similar, though not compatible).
- There seems to be no other contenders.
MessagePack is more widely used and has more mature and well-maintained implementations in all languages of interest.
TODO: Java benchmarks against 2.x binary protocol.
MsgPack Example
packer
.packString("select * from cars where year > ? and seats = ?")
.packArrayHeader(2)
.packInt(2005)
.packInt(2);
Result is 54 bytes:![](/confluence/download/attachments/181310009/MsgPackSqlExample.svg?version=1&modificationDate=1623868464000&api=v2)
![](/confluence/download/attachments/181310009/MsgPackSqlExample.png?version=1&modificationDate=1623868874000&api=v2)
For comparison, current Ignite binary protocol encodes the same data in 71 bytes, and takes twice as much time to do so.
MsgPack Customization
TODO: We need custom resolvers for (at least) UUID, dates, decimals.
Risks and Assumptions
- There is no true random access to fields by name in MsgPack - offsets are not stored, values are written sequentially. Though it is possible to skip values without reading them.
Discussion Links
// TODO
Reference Links
Tickets
key |
summary |
type |
created |
updated |
due |
assignee |
reporter |
priority |
status |
resolution |
JQL and issue key arguments for this macro require at least one Jira application link to be configured
|