Geode's Protobuf-based messages are transmitted in “delimited” form, having the size of the serialized message written in var-int form followed by the serialized message bytes. This is a common practice when Protobuf is used for communications so that you know how many bytes to read from the network when receiving a message.

Unfortunately Protobuf does not have uniform support for sending and receiving “delimited” messages. In some languages you can use a stream object to read the expected size of the message. In other languages you must read some amount of the message into a buffer, then use a decoder to pick out the size bytes and then keep reading from the TCP/IP socket until you have read all of the expected bytes. Then you can decode the Protobuf message.

 

 

Here are options for working with size-delimited messages insome of the more popular languages supported by Protobuf:

In Java you can use the AbstractMessageLite (the class that all messages extend) method readDelimited() to write the message and parseDelimitedFrom() to read the message.

In JavaScript you can use BinaryDecoder and BinaryEncoder to read the size bytes. Since these work on buffers and not streams you must read all of the size bytes into a buffer and then decode the varint32 before you will know how many bytes need to be read to form the complete Protobuf message.

In C++ the writing and reading of size-delimited messages is done with functions described in google/protobuf/util/delimited_message_util.h.

In C# use CodedOutputStream and CodedInputStream. These have message writing/reading size-delimited messages.

In Go use Buffer.DecodeMessage() and Buffer.EncodeMessage().

In Python you must write the var-int delimiters yourself or use the internal methods _VarintBytes(size) and _DecodeVarint32(buffer, position). These return the position in the buffer just after the size bytes.

Here's a Python example. Note that the read-side constructs a new buffer that is limited to reading only the bytes of the message.

size = my_metric.ByteSize()

f.write(_VarintBytes(size))

f.write(my_metric.SerializeToString())

 

msg_len, new_pos = _DecodeVarint32(buf, 0)

msg_buf = buf[new_pos:new_pos+msg_len]

read_metric = metric_pb2.Metric()

read_metric.ParseFromString(msg_buf)

 

In Ruby use ProtocolBuffers::Varint to encode and decode the message size.

Objective C has readDelimited and writeDelimited method variants in GPBMessage.

In PHP there may be something better but if not you can use CodedOutputStream.writeVarInt32() and readVarInt32() to encode and decode the sizes.

 

In cases where there are no stream-oriented means for reading the size directly from the TCP/IP socket you can use the trick of reading bytes until you encounter one having a zero high bit, which is the last byte of the varint.

do {

sizeBytes[index++] = socketStream.readByte()

} while (sizeBytes[index] & 0x80 == 0x80);

decode varint32 from sizeBytes

 

  • No labels