Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: fix typo

...

  • Check if the event is large message by looking forlarge-message: trueheader
    • If it does have `large-message` header:
      • Parse the event and retrieve the ID and the needed useful information on the event.
      • Use the provided PayloadStore implementation to download the original data from the payload-store.
      • Return payload as the final value
    • If it doesn't have the large-message header:
      • Do nothing return the Kafka message as it is

Proposed Changes

1.

...

org.apache.kafka.common.

...

serialization.ComposableSerializer

...

and org.apache.kafka.common.

...

serialization.ComposableDeserializer

Configuration

KeyDescriptionTypeDefaultRequired
value.serializersList of serializers class that implements the `org.apache.kafka.common.serialization.Serializer` interface in order. The first serializer is `Serializer<T>` while the remaining serializers must be `Serializer<byte[]>`.StringNoneNo, If not exist the code will fail back to value.serializer 
value.deserializersList of deserializer classes that implements the `org.apache.kafka.common.serialization.Deserializer` interface in order. The last deserializer must be `Deserializer<T>` while the rest must be `Deserializer<byte[]>`.StringsNoneNo, If not exist the code will fail back to value.deserializer 

2. org.apache.kafka.common.

...

serialization.largemessage.LargeMessageSerializer

Configuration

KeyDescriptionTypeDefaultRequired
large.message.payload.store.classImplementation of org.apache.kafka.common.serialization.largemessage.store.PayloadStore .StringNoneYes
large.message.threshold.bytes The maximum size of the message is considered large.Long1MB (Default of max.message.bytes)No

3.org.apache.kafka.common.

...

serialization.largemessage.LargeMessageDeserializer

Configuration

KeyDescriptionTypeDefaultRequired
large.message.payload.store.classImplementation of org.apache.kafka.common.serialization.largemessage.store.PayloadStore .StringNoneYes
large.message.skip.not.found.error Skip not found error when the payload is not found in the store. This allows the deserializer to skip not-found messages and return empty bytes instead new byte[0] . This is important when external store ttl is smaller than kafka retention.BooleanFALSENo

4.org.apache.kafka.common.

...

serialization.largemessage.PayloadStore

Code Block
languagejava
/**
* The contract for any PayloadStore implementation. 
* And extract them from the provided configs. 
*/
public interface PayloadStore implements Configurable, Closeable, Monitorable {
     /**
     * Publish data into the store.
     *
     * @param data data that will be published to the store.
     * @return {@link PayloadResponse}.
     */
    public abstract PayloadResponse publish(String topic, byte[] data);

    /**
     * Download full data from the store.
     *
     * @param path id of the data's reference in the store
     * @return content of the object as bytes.
     */
    public abstract byte[] download(String path);

    /** 
    * Generate an id for the data's reference in the store. 
    * By default the id is a random UUID however some stores might need more smarter way to calculate its reference id. 
    * In such a case please override this method. 
    * @param data data that will be published to the store.
    public String id(byte[] data) {
       return UUID.randomUUID().toString();
    }
}

5. org.apache.kafka.common.

...

serialization.largemessage.PayloadResponse

Code Block
languagejava
/**
* Response from publish / download from PayloadStore back to the serialization layer
* It contains the final path, response code and the encountered exception if there was any. 
* If PlayloadResponse contains PayloadStoreException with isRetryable flag then it will serialiser will
* retry. 
 */
public class PayloadResponse {
    public final String fullPayloadPath;
    public final PayloadStoreException payloadStoreException;
    /**
     * Construct payload response with response code and payload id.
     */
    public PayloadResponse(String fullPayloadPath) {
        this(fullPayloadPath, null);
    }

    /**
     * Construct payload response with payload id and exception.
     */
    public PayloadResponse(String fullPayloadPath, PayloadStoreException payloadStoreException) {
        this.fullPayloadPath = fullPayloadPath;
        this.payloadStoreException = payloadStoreException;
     }
}

6. org.apache.kafka.common.

...

serialization.largemessage.PayloadStoreException

Code Block
/**
* Exception class that can either be reliable or not
* this helps the serializer/desrializer to decided either to retry or to crash. 
* One subclass will be added is PayloadNotFoundException which is used to indicated if the payload not found 
* This is used by deserializer to skip or not.
**/
public class PayloadStoreException extends RuntimeException {
    /**
     * Constructor PayloadStoreException with message and throwable.
     */
    public PayloadStoreException(String message, Throwable t) {
        super(message, t);
    }

    /**
     * Constructor PayloadStoreException with message.
     */
    public PayloadStoreException(String message) {
        super(message);
    }

    /**
     * Constructor PayloadStoreException with throwable.
     */
    public PayloadStoreException(Throwable t) {
        super(t);
    }
}

...