Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

1. Composable Serializer/Deserializer

Configuration

KeyDescriptionDefaultRequired
value.serializersList of serializers class that implements the `org.apache.kafka.common.serialization.Serializer` interface in order. The first serializer is `Serializer<T>` while the remaining serializers must be `Serializer<byte[]>`.NoneYes
value.deserializersList of deserializer classes that implements the `org.apache.kafka.common.serialization.Deserializer` interface in order. The last deserializer must be `Deserializer<T>` while the rest must be `Deserializer<byte[]>`.NoneYes

2. LargeMessageSerializer

Configuration

KeyDescriptionDefaultRequired
large.message.payload.store.classImplementation of org.apache.kafka.common.serialization.largemessage.store.PayloadStore .NoneYes
large.message.payload.store.timeout.msTimeout for payload store operations. This can't exceedmax.block.msfor Kafka producers ormax.poll.interval.msin Kafka consumer. This is a part of basic configurations for any implementation of org.apache.kafka.common.serialization.largemessage.store.PayloadStore10000No
large.message.payload.store.retry.countNumber of retries for payload store operations. This is a part of basic configurations for any implementation of org.apache.kafka.common.serialization.largemessage.store.PayloadStore5No
large.message.payload.store.retry.max.backoff.msBackoff time between retries for payload store operations. This is a part of basic configurations for any implementation of org.apache.kafka.common.serialization.largemessage.store.PayloadStore10000No
large.message.payload.store.retry.delay.backoff.msDelay time between retries back off for payload store operations. This is a part of basic configurations for any implementation of org.apache.kafka.common.serialization.largemessage.store.PayloadStore100No
large.message.threshold.bytes The maximum size of the message is considered large.1MB (Default of max.message.bytes)No




3. LargeMessageDeserializer

Configuration

KeyDescriptionDefaultRequired
large.message.payload.store.classImplementation of org.apache.kafka.common.serialization.largemessage.store.PayloadStore .NoneYes
large.message.payload.store.timeout.msTimeout for payload store operations. This can't exceedmax.block.msfor Kafka producers ormax.poll.interval.msin Kafka consumer. This is a part of basic configurations for any implementation of org.apache.kafka.common.serialization.largemessage.store.PayloadStore10000No
large.message.payload.store.retry.countNumber of retries for payload store operations. This is a part of basic configurations for any implementation of org.apache.kafka.common.serialization.largemessage.store.PayloadStore5No
large.message.payload.store.retry.max.backoff.msBackoff time between retries for payload store operations. This is a part of basic configurations for any implementation of org.apache.kafka.common.serialization.largemessage.store.PayloadStore10000No
large.message.payload.store.retry.delay.backoff.msDelay time between retries back off for payload store operations. This is a part of basic configurations for any implementation of org.apache.kafka.common.serialization.largemessage.store.PayloadStore100No
large.message.skip.not.found.error Skip not found error when the payload is not found in the store. This allows the deserializer to skip not-found messages and return empty bytes instead. This is important when external store ttl is smaller than kafka retention.FALSENo




4. PayloadStore

Code Block
languagejava
/**
* The contract for any PayloadStore implementation. 
* This parent abstract class will validate the initial configurations that any payload store must have, like large.message.payload.store.timeout.ms, large.message.payload.store.retry.count, large.message.payload.store.retry.max.backoff.ms and large.message.payload.store.retry.delay.backoff.ms. 
* And extract them from the provided configs. 
*/
public abstract class PayloadStore implements Configurable, Closeable {

    Integer maxRetries;
    Integer timeoutMs;
    Long maxBackoffMs;
    Long delayBackoffMs;
    protected Metrics metrics;     
    
    @Override
    public void configure(Map<String, ?> configs) {
        // configure
    }

     /**
     * Publish data into the store.
     *
     * @param data data that will be published to the store.
     * @return {@link PayloadResponse}.
     */
    public abstract PayloadResponse publish(String topic, byte[] data);

    /**
     * Download full data from the store.
     *
     * @param path id of the data's reference in the store
     * @return content of the object as bytes.
     */
    public abstract byte[] download(String path);

    /** 
    * Generate an id for the data's reference in the store. 
    * By default the id is a random UUID however some stores might need more smarter way to calculate its reference id. 
    * In such a case please override this method. 
    * @param data data that will be published to the store.
    public String id(byte[] data) {
       return UUID.randomUUID().toString();
    }
}

...

5.

...

PayloadResponse

Code Block
languagejava
/**
* Response from PayloadReferenceValuepublish represent/ thedownload finalfrom publishedPayloadStore payloadback referenceto intothe Kafka.serialization layer
* It includescontains the fullfinal payload path, onresponse thecode storeand asthe wellencountered asexception theif storethere class used for the publishing.was any. 
* ThisIf allowPlayloadResponse thecontains deserializerPayloadStoreException towith throwisRetryable aflag morethen informativeit errorswill if the original payload store class doesn't match the one setup by the consumer client. 
*/
{
  "apiKey": 0,
  "type": "payload-reference",
  "name": "PayloadReferenceValue",
  // Version 0 KIP-1159.
  "validVersions": "0",
  "fields": [
    { "name": "fullPayloadPath", "versions": "0+", "type": "string", "about": "The full path to access payload as string."},
    { "name": "payloadStoreClass", "versions": "0+", "type": "string","about": "The used payload store class path to upload payload."}
  ]
}

7. PayloadResponse

Code Block
languagejava
/**
* Response from publish / download from PayloadStore back to the serialization layer
* It contains the final path, response code and the encountered exception if there was any. 
serialiser will
* retry. 
* If the responseCode 404 the deserialiser will skip the error if large.message.skip.not.found.error set to true.
 */
public class PayloadResponse {
    public final int responseCode;
    public final String pathfullPayloadPath;
    public final PayloadStoreException payloadStoreException;
    /**
     * Construct payload response with response code and payload id.
     */
    public PayloadResponse(int responseCode, String pathfullPayloadPath) {
        this(responseCode, pathfullPayloadPath, null);
    }

    /**
     * Construct payload response with response code, payload id and exception.
     */
    public PayloadResponse(int responseCode, String pathfullPayloadPath, PayloadSto
reException payloadStoreException) {
        this.responseCode = responseCode;
        this.fullPayloadPath = pathfullPayloadPath;
        this.payloadStoreException = payloadStoreException;
     }
}

...

6. PayloadStoreException

Code Block
/**
* Exception class that can either be reliable or not
* this helps the serializer/desrializer to decided either to retry or to crash. 
* One subclass will be added is PayloadNotFoundException which is used to indicated if the payload not found 
* This is used by deserializer to skip or not.
**/
public class PayloadStoreException extends RuntimeException {
    protected boolean isRetryable = truefalse;

    /**
     * Constructor PayloadStoreException with message and throwable.
     */
    public PayloadStoreException(String message, Throwable t) {
        super(message, t);
    }

    /**
     * Constructor PayloadStoreException with message.
     */
    public PayloadStoreException(String message) {
        super(message);
    }

    /**
     * Constructor PayloadStoreException with throwable.
     */
    public PayloadStoreException(Throwable t) {
        super(t);
    }

    /**
     * Constructor PayloadStoreException with message, throwable and if it is retryable or not.
     */
    public PayloadStoreException(String message, Throwable t, boolean retryable) {
        this(message, t);
        isRetryable = retryable;
    }

    /**
     * return whether the exception is retryable or not.
     */
    public boolean isRetryable() {
        return isRetryable;
    }
}

...