Versions Compared

Old Version 38

changes.mady.by.user Pavel Tupitsyn

Saved on

compared with

New Version 39

changes.mady.by.user Pavel Tupitsyn

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Thin client protocol (handshake, message format, etc) will be designed in a separate IEP. Here we only discuss a mechanism to serialize user and system data: primitive and compound values, such as cache entries, configuration objects, and so on.

Description

...

Use MsgPack format in the Ignite 3.0 thin client protocol.

MsgPack Example

Code Block
languagejava
titleSQL request (query text + arguments array
packer
        .packString("select * from cars where year > ? and seats = ?")
        .packArrayHeader(2)
        .packInt(2005)
        .packInt(2);

Result is 54 bytes:Image Added

Image Added

For comparison, current Ignite binary protocol encodes the same data in 71 bytes, and takes twice as much time to do so (see benchmarks linked below).

Why MsgPack

The goal is to find an existing serialization format that satisfies the following requirements:

...

MessagePack is more widely used and has more mature and well-maintained implementations in all languages of interest.

TODO: Java benchmarks against 2.x binary protocol.

MsgPack Example

Benchmarks

  • Code is linked below
  • MsgPack is always faster on primitive values
  • MsgPack is more compact because of varints everywhere
  • Ignite is faster on POJOs, because MsgPack uses Jackson integration to handle objects, which is very configurable and nice, but comes at a cost. We can develop our own implementation if needed.
No Format
 * Benchmark                                                       Mode  Cnt         Score        Error  Units
 * JmhBinaryMarshallerMsgPackBenchmark.writePrimitivesMsgPackRaw  thrpt   10  16834154.556 ± 85624.143  ops/s
 * JmhBinaryMarshallerMsgPackBenchmark.writePrimitivesIgnite      thrpt   10  12702562.838 ± 248094.068  ops/s
 *
 * JmhBinaryMarshallerMsgPackBenchmark.writePojoIgnite            thrpt   10  11590924.790 ±  42061.734  ops/s // Full footers
 * JmhBinaryMarshallerMsgPackBenchmark.writePojoMsgPack           thrpt   10   5386377.535 ±  33835.097  ops/s // Fields with names
 * JmhBinaryMarshallerMsgPackBenchmark.writePojoMsgPack2          thrpt   10   8505961.494 ± 465369.449  ops/s // Fields without names
 *
 * JmhBinaryMarshallerMsgPackBenchmark.readPrimitivesIgnite       thrpt   10  19873521.096 ± 545779.558  ops/s
 * JmhBinaryMarshallerMsgPackBenchmark.readPrimitivesMsgPack
Code Block
languagejava
titleSQL request (query text + arguments array
packer
       thrpt .packString("select * from cars where year > ? and seats = ?")
        .packArrayHeader(2)
  10  29235107.372 ±  85371.004  ops/s
 *
 * JmhBinaryMarshallerMsgPackBenchmark.readPojoIgnite             thrpt   10  8437054.066 ± 104476.415  ops/s
 * JmhBinaryMarshallerMsgPackBenchmark.readPojoMsgPack            thrpt   10  6292876.474 ±  73356.915  ops/s
 *
 * JmhBinaryMarshallerMsgPackBenchmark.writeSqlQueryIgnite        .packInt(2005)
  thrpt   10   5756908.336 ±  42079.083  ops/s
 * JmhBinaryMarshallerMsgPackBenchmark.writeSqlQueryMsgPack      .packInt(2);

Result is 54 bytes:Image Removed

Image Removed

For comparison, current Ignite binary protocol encodes the same data in 71 bytes, and takes twice as much time to do so (see benchmarks linked below).

...

 thrpt   10  12380076.956 ± 150712.634  ops/s


MsgPack Custom Types

TODO: We need custom resolvers for (at least) UUID, dates, decimals.

...