...
In some use-cases we might need a way to be able to chain of Kafka serializer/deserializer before applying large message serializer for example need to apply schema or special format like avro or protobuf. We could just create a single org.apache.kafka.common.serialization.largemessage.Serializer that implemented the Kafka Serializer interface, but we would need to create the different versions (AvroSerializer, ProtobufSerializer) with their support for the schema.
Instead the KIP proposing a composable serializer , that still implements the Kafka Serializer interface, but allows concatenating several serializers to perform what we are looking for here.
This will be enabled by new configs in Producer/Consumer side
2. org.apache.kafka.common.serialization.largemessage.Serializer
...
- Check if the event is large message by looking for
large-message: trueheader
- If it does have `
large-message` header:
- Parse the event and retrieve the ID and the needed useful information on the event.
- Use the provided PayloadStore implementation to download the original data from the payload-store.
- Return payload as the final value
- If it doesn't have the
large-message header:- Do nothing return the Kafka message as it is
Proposed Changes
1
...
. New ProducerConfig to support Composable Serializer
Configuration
| Key | Description | Type | Default | Required |
| value.serializers | List of serializers class that implements the `org.apache.kafka.common.serialization. |
...
| Serializer` interface in order. The first serializer is `Serializer<T>` while the remaining serializers must be `Serializer<byte[]>`. | List<Serializer> | None | No, If not exist the code will fail back to value.serializer. Can't exist with value.serializer |
| key.serializers | List of serializers class that implements the `org.apache.kafka.common.serialization.Serializer` interface in order. The first serializer is `Serializer<T>` while the remaining serializers must be `Serializer<byte[]>`. | List<Serializer> | None | No, If not exist the code will fail back to key.serializer. Can't exist with key.serializer |
2. New ConsumerConfig to support Composable Deserializer
Configuration
| Key | Description | Type | Default | Required |
| value. |
serializers serializers class | deserializer classes that implements the `org.apache.kafka.common.serialization. |
Serializer` | Deserializer` interface in order. The |
first serializer is `Serializer<T>` | last deserializer must be `Deserializer<T>` while the |
remaining serializers `Serializer<byteList<Serializer>| List<Deserialzer> | None | No, If not exist the code will fail back to value. |
serializerdeserializer. Can't exist with |
serializervalue| key.deserializers | List of deserializer classes that implements the `org.apache.kafka.common.serialization.Deserializer` interface in order. The last deserializer must be `Deserializer<T>` while the rest must be `Deserializer<byte[]>`. | List<Deserialzer> | None | No, If not exist the code will fail back to |
valuekey.deserializer. Can't exist with |
value...
3. org.apache.kafka.common.serialization.largemessage.Serializer<byte[]>
Configuration
| Key | Description | Type | Default | Required |
|---|
large.message.payload.store.class | Implementation of org.apache.kafka.common.serialization.largemessage.store.PayloadStore . | Class | None | Yes |
large.message.threshold.bytes | The maximum size of the message is considered large. | Long | 1MB (Default of max.message.bytes) | No |
...
4.org.apache.kafka.common.serialization.largemessage.Deserializer<byte[]>
Configuration
| Key | Description | Type | Default | Required |
|---|
large.message.payload.store.class | Implementation of org.apache.kafka.common.serialization.largemessage.store.PayloadStore . | Class | None | Yes |
...
5.org.apache.kafka.common.serialization.largemessage.PayloadStore
| Code Block |
|---|
|
/**
* An interface for publishing and downloading serialised data to/from payload store.
* The store config will passed down from the original config of the Kafka producer/consumer client.
*/
public interface PayloadStore implements Configurable, Closeable, Monitorable {
/**
* Publish data into the store.
*
* @param data data that will be published to the store.
* @return full path to object in the store.
* @throw PayloadStoreException in case failed to publish to the store.
*/
String publish(String topic, byte[] data) throw PayloadStoreException;
/**
* Download full data from the store.
*
* @param fullPath of the data's reference in the store for example `remote_store/topic_name/<record_random_uuid>`
* @return content of the object as bytes.
* @throw PayloadStoreException
*/
byte[] download(String fullPath) throw PayloadStoreException;
/**
* Generate an id for the data's reference in the store (Not the full path in the store).
* By default the id is a random UUID however some stores might need more smarter way to calculate
* its reference id based on the data itself. In such a case please override this method.
* @param data data that will be published to the store.
* @return object id for example `record_random_uuid`
*/
default String id(byte[] data) {
return UUID.randomUUID().toString();
}
} |
...
6. org.apache.kafka.common.serialization.largemessage.PayloadStoreException
| Code Block |
|---|
/**
* Exception class that represent exceptions during interaction with the store.
* this helps the Payload store to decided either to retry or to crash.
* The final Serializer and Deserializer will propagate this as SerializationException to client.
**/
public class PayloadStoreException extends RuntimeException {
/**
* Constructor PayloadStoreException with message and throwable.
*/
public PayloadStoreException(String message, Throwable t) {
super(message, t);
}
/**
* Constructor PayloadStoreException with message.
*/
public PayloadStoreException(String message) {
super(message);
}
/**
* Constructor PayloadStoreException with throwable.
*/
public PayloadStoreException(Throwable t) {
super(t);
}
}
|
...