...

1. Use an attribute bit as a tombstone flag on the core message format.
2. This "tombstone" attribute bit is only used by the broker when a topic is configured for compaction. 
   1. If the topic is not configured for compaction it will be set and on the record and available to be read but will not cause message deletion by the broker.
   2. This will need to be added to end user documention.

3. Add a tombstone boolean field to ProducerRecord and ConsumerRecord. 

4. Add constructor param on ProducerRecord to set the tombstone on creation of a Record
5. Add accessor methods on the Producer/ConsumerRecord boolean isTombstone()
6. Add ProduceRequest/ProduceResponse V4 which V3 which uses the new message format.
7. Add FetchRequest/FetchResponse V4 V3 which uses the new message format.
   
   

...

Compatibility, Deprecation, and Migration Plan

• Migration plan :

  • Currently the code version is 0.10.1, message.format.version = 0.10.1, IBP = 0.10.1.
  • Create internal ApiVersion 0.10.2-IV0 which uses ProduceRequest V4 V3 (magic byte = 2) and FetchRequest V4 V3 (magic byte =2).
  • We first upgrade the code to support ApiVersion 0.10.2-IV0 with a rolling upgrade.
  • Do a rolling bounce and set the IBP = 0.10.2
  • Upgrade clients to start producing and consuming using ProduceRequest V4 V3 (magic byte = 2) and FetchRequest V4 V3 (magic byte =2).
  • At this point if the broker receives ProduceRequest V4 V3 from producer with the tombstone bit set in the message, it will down convert it to set it to null. ProduceRequest V3 V2 will work as it does today.
  • At this point if the broker receives FetchRequest V3 V2 from consumer, we will not loose zero copy because the message is down converted. If the broker receives FetchRequest V4V3, it will still work as the consumer will be able to understand the tombstone bit.
  • When all the clients have upgraded, bump up the message.format.version to 0.10.2 on the broker. We should be careful here to see that almost all consumers are upgraded since at this point if broker receives a FetchRequest V3V2, we might loose zero copy on the broker.
    • To note on a downgrade for an non-upgraded consumer, the downgrade would make the value null for a tombstone message.
  • From log compaction point of view :
  • If the magic byte on message is 0, the broker should use the null value for log compaction.
  • If the magic byte on message is 1, the broker should use the null value for log compaction.
  • If the magic byte on message is 2, the broker should use the tombstone bit for log compaction.
  • NOTE : With the new version of producer client using ProduceRequest V4 V3 (magic byte = 2), a non tombstone (tombstone bit not set) and null value should not be allowed as the older version of consumer using FetchRequest V3 V2 will think of this as a tombstone when its actually not.
• Client Compatibility:
  • If simply upgrading client, but no changes to code/flow (e.g. still sending null tombstones via existing constructors), we do not expect any app changes needed.
  • Producer
    • A new constructors for ProducerRecord will add an extra parameter tombstone: 
    • The old constructors for ProducerRecord without this parameter will be kept but updated so that their default behaviour if setting null value will be the tombstone will be set to true to keep existing behaviour.
  • Consumer
    • The ConsumerRecord will expose a new method isTombstone will be exposed
    • Existing flows that use null tombstones and continue to do so will not be affected as the value for a tombstone will still be null
      • As such simply upgrading clients and not the flow behaviour there will be node change needed.
      • It is recommended that their logic is updated to use the isTombstone for clean-ness.
    • New/Amended flows created where a producer may send non-null tombstones, the new consumer code will need to use isTombstone.

Rejected Alternatives

Do nothing KIP-82 would remove the immediate issue.

...