Page History

...

For reference, the current message format (v1) is the following:

MessageSet => [Offset MessageSize Message]
 Offset => int64
 MessageSize => int32

Message => Crc Magic Attributes Timestamp Key Value
 Crc => int32
 Magic => int8
 Attributes => int8
 Timestamp => int64
 Key => bytes
 Value => bytes

...

A message set is a sequence of messages. To support compression, we currently play a trick with this format and allow the compressed output of a message set to be embedded in the value field of another message (a.k.a., the “wrapper message”). In this design, we propose to extend this concept to non-compressed messages and to decouple the schema for the message wrapper (which contains the compressed message set). This allows us to maintain a separate set of fields at the message set level and avoid some costly redundancy:

...

MessageSet => 
  Offset => int64
  Length => int32
  CRC => int32
  Magic => int8  /* bump up to “2” */
  Attributes => int16
  OffsetDelta => int32 {NEW}
  MaxTimestamp => int64 {NEW}
  PID => int64 {NEW}
  Epoch => int16 {NEW}
  Sequence => int32 {NEW}
  Messages => Message1, Message2, … , MessageN {NEW}

Message => {ALL FIELDS NEW}
  Length => uintVar
  Attributes => int8
  Timestamp => int64
  OffsetDelta => uintVar
  KeyLen => uintVar [OPTIONAL]
  Key => data [OPTIONAL]
  Value => data [OPTIONAL]
  CRC => int32

...