THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
We also require additional authorization to produce transactional data. This can be used to minimize the risk of an “endless transaction attack,” in which a malicious producer writes transactional data without corresponding COMMIT or ABORT markers in order to prevent the LSO from advancing and consumers from making progress. We can use the ProducerTransactionalId resource introduced above to ensure that the producer is authorized to write transactional data . The as the producer’s TransactionalId is also included in the ProduceRequest schema.
...
| Code Block | ||
|---|---|---|
| ||
MessageSet =>
Offset => int64
Length => int32
CRC => int32
Magic => int8 /* bump up to “2” */
Attributes => int16
OffsetDelta => int32 {NEW}
MaxTimestamp => int64 {NEW}
PID => int64 {NEW}
Epoch => int16 {NEW}
Sequence => int32 {NEW}
Messages => Message1, Message2, … , MessageN {NEW}
Message => {ALL FIELDS NEW}
Length => uintVar
Attributes => int8
Timestamp => int64
OffsetDelta => uintVar
KeyLen => uintVar [OPTIONAL]
Key => data [OPTIONAL]
Value => data [OPTIONAL]
CRC => int32
|
The ability to store some fields only at the message set level allows us to conserve space considerably when batching messages into a message set. For example, there is no need to write the PID within each message since it will always be the same for all messages within each message set. In addition, by separating the message level format and message set format, now we can also use variable-length types for the inner (relative) offsets and save considerably over a fixed 8-byte field size.
...