Page History

...

• Stream the data out of Kafka and perform Event Sourcing there
  • This would mean creating an in-house solution, which makes Kafka irrelevant in the design, and so its best left as a last-approach in case no solution is found on Kafka-side
• Guarantee insertion order on the producer
  • Not viable as keeping this logic synchronized greatly reduces the event throughput
• Check the version before sending the event to Kafka
  • Similar to the previous point, though it adds extra complexity as race-conditions may arise when attempting to compare
• Caching the record version as a byte array and perform the comparisons between records using a lexicographic byte array comparator
  • This adds greater flexibility on the client side, but allowing a variable byte array size to be used raises concerns about memory usage by the cache
• Always search the headers for a key matching whatever is configured, so if a header "timestamp" exists then it could be used by the compaction mechanism
  • This introduces backwards compatibility issues, as any headers are allowed at the moment without impacting compaction.
  • Even if ignoring the previous point, this may cause API issues as, for example, the topic may be designed with "offset" compaction, which makes it unclear if the Producer should then provide a header "offset" or if the internal offset is meant to be used.