...

Recommendation: Set TTL duration to exceed your Kafka topic retention period by a reasonable buffer (e.g., 10-20%) to ensure payload availability throughout the message lifecycle while preventing indefinite storage growth.

• Partial Failure Handling: If payload storage succeeds but Kafka produce fails, the payload will remain in storage until TTL expires. This is acceptable as it only affects storage costs, not correctness. If Kafka message is consumed but payload download fails, the large.message.skip.not.found.error  configuration determines behavior.

• Critical Timing Constraints: The total timeout for payload store operations (including all retries) cannot exceed `max.block.ms` for Kafka producers or `max.poll.interval.ms` for Kafka consumers. This is a fundamental requirement for any implementation of org.apache.kafka.common.serialization.largemessage.store.PayloadStore.

  • Exceeding these limits will cause producer blocking or consumer rebalancing issueswill cause producer blocking or consumer rebalancing issues.

• Memory Constrains: Large messages will consume memory during serialization so ensure heap size can accommodate your largest expected message. ConsiderPayloadStoreimplementationsthatsupportcompressiontoreducestoragefootprint.

Compatibility, Deprecation, and Migration Plan

...

• We have rejected implementing the chunking pattern due to its many potential edge cases and complexity added to the consumer side. A peak of those complexities can be explored more deeply in the LinkedIn presentation.
• Implement this as a separate project outside of Apache Kafka as this seems to be a pattern that needs more use cases and it would be better to have this in Apache Kafka as native implementation instead of a separate project
• Implement the AsyncPayloadStore: This is bit tricky as the Serializer in Kafka is a blocker stepdoesn'tprovideanybenefitsinceKafka'sSerializerinterfaceissynchronous/blocking-anyasyncoperationswouldneedtoimmediatelycall.get(),negatingtheasyncbenefits. Also the reality is that most of use-cases that needs more > 1MB messages:
  

  • Are dealing with bulk data (collections, arrays)

  • Can afford the memory cost during brief serialization

  • Run on appropriately sized JVMs

...