THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
MessageAndOffset => Offset MessageSize Message Offset => int64 MessageSize => int32 Message => Crc MagicByte Attributes Timestamp KeyLength Key HeadersLength Headers ValueLength Value Crc => int32 MagicByte => int8 Attributes => int8 <---------------------- Use Bit 5 as boolean flag for 'isTombstone' flag Timestamp => int64 KeyLength => int32 Key => bytes ValueLength => int32 Value => bytes |
LogCleaner
Update method "shouldRetainMessage" to also look at attribute bit 5 for tombstone marker
...
Migration plan :
- Currently the code version is 0.10.1, message.format.version = 0.10.1, IBP = 0.10.1.
- Create internal ApiVersion 0.10.2-IV0 which uses ProduceRequest V3 V4 (magic byte = 2) and FetchRequest V3 V4 (magic byte =2).
- We first upgrade the code to support ApiVersion 0.10.2-IV0 with a rolling upgrade.
- Do a rolling bounce and set the IBP = 0.10.2
- Upgrade clients to start producing and consuming using ProduceRequest V3 V4 (magic byte = 2) and FetchRequest V3 V4 (magic byte =2).
- At this point if the broker receives ProduceRequest V3 V4 from producer with the tombstone bit set in the message, it will down convert it to set it to null. ProduceRequest V2 V3 will work as it does today.
- At this point if the broker receives FetchRequest V2 V3 from consumer, we will not loose zero copy because the message is down converted. If the broker receives FetchRequest V3V4, it will still work as the consumer will be able to understand the tombstone bit.
- When all the clients have upgraded, bump up the message.format.version to 0.10.2 on the broker. We should be careful here to see that almost all consumers are upgraded since at this point if broker receives a FetchRequest V2V3, we might loose zero copy on the broker.
- From log compaction point of view :
- If the magic byte on message is 0, the broker should use the null value for log compaction.
- If the magic byte on message is 1, the broker should use the null value for log compaction.
- If the magic byte on message is 2, the broker should use the tombstone bit for log compaction.
- NOTE : With the new version of producer client using ProduceRequest V3 V4 (magic byte = 2), a non tombstone (tombstone bit not set) and null value should not be allowed as the older version of consumer using FetchRequest V2 V3 will think of this as a tombstone when its actually not.
...