Page History

...

• Check if the size of the data (bytes) needs to serialize is larger than the configured threshold.
  • If it is large than the provided threshold (`large.message.threshold.bytes`):
    • Use the provided PayloadStore implementation to publish the large message into payload store, generating an id which is the reference to access this later.
    • Encapsulate that ID into a simple Kafka event using a structured format.
    • Pass the Pass payload Id as new Kafka Event down to Kafka 
    • Add alarge-message: trueheader
  • If it’s not large then provided threshold (large.message.threshold.bytes):
    • Do nothing, pass the data as it is.

...

• TTL Configuration Risk: If the payload store owner doesn't configure an appropriate TTL that aligns with Kafka topic retention, the payload store may grow indefinitely. This occurs because objects remain in storage even after Kafka no longer references them, leading to unnecessary storage costs.

• TTL Too Short Risk: If the TTL is set too aggressively (shorter than needed), Kafka references may point to objects that no longer exist in the payload store. When this happens:

  • Consumers will encounter NOT_FOUND errors

  • To prevent blocking behavior, consumers should capture SerializationException::getCause and decide what to do if the exception is PayloadException/PayloadNotFoundException.

  • This allows graceful handling of missing payload references

• TTL and usage of Compacted Topic: for applications that do use compacted topics with large payloads, the PayloadStore implementation should handle this by:

  • Consistent ID Generation:PayloadStore implementations should use deterministic IDs based on message content or metadata (rather than random UUIDs) so that identical payloads can reuse the same storage object, reducing storage costs.

  • TTL Strategy:

    • Since compacted topics can retain data indefinitely, users must choose between setting a business-appropriate TTL or accepting indefinite storage costs.

    • Topics with cleanup.policy=compact,delete  will eventually remove old data, so standard TTL approaches work normally.

Recommendation: Set TTL duration to exceed your Kafka topic retention period by a reasonable buffer (e.g., 10-20%) to ensure payload availability throughout the message lifecycle while preventing indefinite storage growth.

...