Page History

...

• Enhance log compaction to support more than just offset comparison, so the insertion order isn't always dictating which records to keep (in effect, allowing for a form of OCC);
• The current behavior should remain as the default in order to minimize impact on already existing clients and avoid any migration efforts;
• New Configurations:
  
  Global:
  • "log.cleaner.compaction.strategy"
    • The active compaction strategy to use;
    • Accepts values "offset", "timestamp" and "header", allowing for further strategies to be added in the future as needed;
  • "log.cleaner.compaction.strategy.header"
    
    • Configuration sub-set to use when the strategy is set to "header";
  • Topic:
    • "compaction.strategy"
      • Represents the same as "log.cleaner.compaction.strategy", but for a specific topic;
    • "compaction.strategy.header"
      • Represents the same as "log.cleaner.compaction.strategy.header", but for a specific topic;
• Compaction Strategies:
  • "offset"
    • The previous behavior is active, compacting the logs purely based on offset;
    • Also used when the configuration is either empty or not present, making this the default strategy;
  • "timestamp"
    • The record timestamp will be used to determine which record to keep, in a 'keep-highest' approach;
    • When both records being compared contain an equal timestamp, then the record with the highest offset will be kept;
  • "header"
    • Searches the record for a header key that matches the configured value on "compaction.strategy.header";
    • If the "compaction.strategy.header" configuration is not set (or is blank), then the compaction strategy will fallback to "offset";
    • If a header key that matches the configuration exists, then the header value (which must be of type "long") will be used to determine which record to keep, in a 'keep-highest' approach;
    • If both records being compared do not have a matching header key, then the record with the highest offset will be kept;
    • If both records being compared contain an equal header value, then the record with the highest offset will be kept;
    • If only one of the records being compared has a matching header, then this record is kept, as the other record is considered to be anomalous;

...

• Stream the data out of Kafka and perform Event Sourcing there
  • This would mean creating an in-house solution, which makes Kafka irrelevant in the design, and so its best left as a last-approach in case no solution is found on Kafka-side
• Guarantee insertion order on the producer
  • Not viable as keeping this logic synchronized greatly reduces the event throughput
• Check the version before sending the event to Kafka
  • Similar to the previous point, though it adds extra complexity as race-conditions may arise when attempting to compare
• Caching the record version as a byte array and perform the comparisons between records using a lexicographic byte array comparator
  • This adds greater flexibility on the client side, but allowing a variable byte array size to be used raises concerns about memory usage by the cache
• Always search the headers for a key matching whatever is configured, so if a header "timestamp" exists then it could be used by the compaction mechanism
  • This introduces backwards compatibility issues, as any headers are allowed without this change and the compaction is not affected at all.
  • Even if ignoring the previous point, this may cause API issues as, for example, the topic may be designed with "offset" compaction, which makes it unclear if the Producer should then provide a header "offset" or if the internal offset is meant to be used.
• Provide the configuration for the individual topics
  • None of the configurations for log compaction are available at topic level, so adding it there is not a part of this KIP