Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

In some use-cases we might need a way to be able to chain of Kafka serializer/deserializer before applying large message serializer for example need to apply schema or special format like avro or protobuf. We could just create a single org.apache.kafka.common.serialization.largemessage.Serializer that implemented the Kafka Serializer interface, but we would need to create the different versions (AvroSerializer, ProtobufSerializer) with their support for the schema.
Instead the KIP proposing a composable serializer , that still implements the Kafka Serializer interface, but allows concatenating several serializers to perform what we are looking for here. 

This will be enabled by new configs in Producer/Consumer side 

2. org.apache.kafka.common.serialization.largemessage.Serializer

...

  • Check if the event is large message by looking forlarge-message: trueheader
    • If it does have `large-message` header:
      • Parse the event and retrieve the ID and the needed useful information on the event.
      • Use the provided PayloadStore implementation to download the original data from the payload-store.
      • Return payload as the final value
    • If it doesn't have the large-message header:
      • Do nothing return the Kafka message as it is

Proposed Changes

1

...

.  New ProducerConfig to support Composable Serializer

Configuration

KeyDescriptionTypeDefaultRequired
value.serializersList of serializers class that implements the `org.apache.kafka.common.serialization.

...

Serializer` interface in order. The first serializer is `Serializer<T>` while the remaining serializers must be `Serializer<byte[]>`.List<Serializer>NoneNo, If not exist the code will fail back to value.serializer. Can't exist with  value.serializer
key.serializersList of serializers class that implements the `org.apache.kafka.common.serialization.Serializer` interface in order. The first serializer is `Serializer<T>` while the remaining serializers must be `Serializer<byte[]>`.List<Serializer>NoneNo, If not exist the code will fail back to key.serializer. Can't exist with key.serializer


2. New ConsumerConfig to support Composable Deserializer

Configuration

KeyDescriptionTypeDefaultRequired
value.
serializers
deserializersList of
serializers class
deserializer classes that implements the `org.apache.kafka.common.serialization.
Serializer`
Deserializer` interface in order. The
first serializer is `Serializer<T>`
last deserializer must be `Deserializer<T>` while the
remaining serializers
rest must be
`Serializer<byte
`Deserializer<byte[]>`.
List<Serializer>
List<Deserialzer>NoneNo, If not exist the code will fail back to value.
serializer
deserializer. Can't exist with
 
value.
serializer
deserializer
value
key.deserializersList of deserializer classes that implements the `org.apache.kafka.common.serialization.Deserializer` interface in order. The last deserializer must be `Deserializer<T>` while the rest must be `Deserializer<byte[]>`.List<Deserialzer>NoneNo, If not exist the code will fail back to
value
key.deserializer. Can't exist with
value
key.deserializer

...

3. org.apache.kafka.common.serialization.largemessage.Serializer<byte[]>

Configuration

KeyDescriptionTypeDefaultRequired
large.message.payload.store.classImplementation of org.apache.kafka.common.serialization.largemessage.store.PayloadStore .ClassNoneYes
large.message.threshold.bytes The maximum size of the message is considered large.Long1MB (Default of max.message.bytes)No

...

4.org.apache.kafka.common.serialization.largemessage.Deserializer<byte[]>

Configuration

KeyDescriptionTypeDefaultRequired
large.message.payload.store.classImplementation of org.apache.kafka.common.serialization.largemessage.store.PayloadStore .ClassNoneYes

...

5.org.apache.kafka.common.serialization.largemessage.PayloadStore

Code Block
languagejava
/**
* An interface for publishing and downloading serialised data to/from payload store.
* The store config will passed down from the original config of the Kafka producer/consumer client. 
*/
public interface PayloadStore implements Configurable, Closeable, Monitorable {
     /**
     * Publish data into the store.
     *
     * @param data data that will be published to the store.
     * @return full path to object in the store.
     * @throw PayloadStoreException in case failed to publish to the store.
     */
     String publish(String topic, byte[] data) throw PayloadStoreException;

    /**
     * Download full data from the store.
     *
     * @param fullPath of the data's reference in the store for example `remote_store/topic_name/<record_random_uuid>`
     * @return content of the object as bytes.
     * @throw PayloadStoreException 
     */
     byte[] download(String fullPath) throw PayloadStoreException;

    /** 
    * Generate an id for the data's reference in the store (Not the full path in the store). 
    * By default the id is a random UUID however some stores might need more smarter way to calculate 
    * its reference id based on the data itself. In such a case please override this method. 
    * @param data data that will be published to the store.
    * @return object id for example `record_random_uuid`
	*/
    default String id(byte[] data) {
       return UUID.randomUUID().toString();
    }
}

...

6. org.apache.kafka.common.serialization.largemessage.PayloadStoreException

Code Block
/**
* Exception class that represent exceptions during interaction with the store.
* this helps the Payload store to decided either to retry or to crash. 
* The final Serializer and Deserializer will propagate this as SerializationException to client.
**/
public class PayloadStoreException extends RuntimeException {
    /**
     * Constructor PayloadStoreException with message and throwable.
     */
    public PayloadStoreException(String message, Throwable t) {
        super(message, t);
    }

    /**
     * Constructor PayloadStoreException with message.
     */
    public PayloadStoreException(String message) {
        super(message);
    }

    /**
     * Constructor PayloadStoreException with throwable.
     */
    public PayloadStoreException(Throwable t) {
        super(t);
    }
}

...