You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 4 Next »

This page is meant as a template for writing a KIP. To create a KIP choose Tools->Copy on this page and modify with your content and replace the heading with the next KIP number and a description of your issue. Replace anything in italics with your own description.

Status

Current state: Under Discussion

Discussion thread: here

JIRA: here

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

This KIP tries to address the following issues in Kafka.

Compaction currently requires a null value for it to mark as a tombstone for deletion. This is covered in KIP-82 as an issue that headers could resolve but is being separated out as is an independent issue in its own right.

This causes issues where organization use message envelopes or wrappers or where a business use case requires delivery of some information on the delete.

This became very apparent on the discussion thread for KIP-82 found here and here, where it has so far been consensus that this issue should be separated out and dealt with separately. 

Public Interfaces

This KIP has the following public interface changes:

  1. Use an attribute bit as a tombstone flag on the core message format.

  2. Add a tombstone boolean field to ProducerRecord and ConsumerRecord. 

  3. Add accessor methods on the Producer/ConsumerRecord void setTombstone(boolean isTombstone) and a boolean isTombstone()
  4. Add ProduceRequest/ProduceResponse V4 which uses the new message format.
  5. Add FetchRequest/FetchResponse V4 which uses the new message format.

For more detail information of the above changes, please refer to the Proposed Changes section.

Proposed Changes

Wire protocol change - use attribute bit5 as flag for tombstone boolean flag. 

  • The attributes flag bit is used to keep the message size the same as before.

 

MessageAndOffset => Offset MessageSize Message
  Offset => int64  
  MessageSize => int32
   
  Message => Crc MagicByte Attributes Timestamp KeyLength Key HeadersLength Headers ValueLength Value
    Crc => int32
    MagicByte => int8  <---------------------- Bump up magic byte to 2
    Attributes => int8 <---------------------- Use Bit 5 as boolean flag for isTombstone glad
    Timestamp => int64
    KeyLength => int32
    Key => bytes
    ValueLength => int32
    Value => bytes

 

LogCleaner

Update method "shouldRetainMessage" to also look at attribute bit 5 for tombstone marker

Compatibility, Deprecation, and Migration Plan

  • Tombstone feature will be backwards compatible
    • Message version migration would be handled as like in KIP-32
    • Bit flag would be default false 
    • Logic would base on current behavior of null value or if tombstone flag set to true, as such wouldn't impact any existing flows simply allow new producers to make use of the feature.

Rejected Alternatives

If there are alternative ways of accomplishing the same thing, what were they? The purpose of this section is to motivate why the design is the way it is and not some other way.

  • No labels