Versions Compared

Old Version 18

changes.mady.by.user Unknown User (omnia)

Saved on

compared with

New Version 19

changes.mady.by.user Unknown User (omnia)

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  • We have rejected implementing the chunking pattern due to its many potential edge cases and complexity added to the consumer side. A peak of those complexities can be explored more deeply in the LinkedIn presentation.
  • Implement this as a separate project outside of Apache Kafka as this seems to be a pattern that needs more use cases and it would be better to have this in Apache Kafka as native implementation instead of a separate project
  • Implement the AsyncPayloadStore: This is bit tricky as the Serializer in Kafka is a blocker step. Also the reality is that most of use-cases that needs more > 1MB messages:

    • Are dealing with bulk data (collections, arrays)

    • Can afford the memory cost during brief serialization

    • Run on appropriately sized JVMs

              Usecases with truly memory-constrained environments probably shouldn't be sending GB-sized messages through Kafka anyway.