DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
...
- Check if the event is large message by looking for
large-message: trueheader- If it does have `
large-message` header:- Parse the event and retrieve the ID and the needed useful information on the event.
- Use the provided PayloadStore implementation to download the original data from the payload-store.
- Return payload as the final value
- If it doesn't have the
large-messageheader:- Do nothing return the Kafka message as it is
- If it does have `
Proposed Changes
1.
...
org.apache.kafka.common.
...
serialization.ComposableSerializer
...
and org.apache.kafka.common.
...
serialization.ComposableDeserializer
Configuration
| Key | Description | Type | Default | Required |
| value.serializers | List of serializers class that implements the `org.apache.kafka.common.serialization.Serializer` interface in order. The first serializer is `Serializer<T>` while the remaining serializers must be `Serializer<byte[]>`. | String | None | No, If not exist the code will fail back to value.serializer |
| value.deserializers | List of deserializer classes that implements the `org.apache.kafka.common.serialization.Deserializer` interface in order. The last deserializer must be `Deserializer<T>` while the rest must be `Deserializer<byte[]>`. | Strings | None | No, If not exist the code will fail back to value.deserializer |
2. org.apache.kafka.common.
...
serialization.largemessage.LargeMessageSerializer
Configuration
| Key | Description | Type | Default | Required |
|---|---|---|---|---|
large.message.payload.store.class | Implementation of org.apache.kafka.common.serialization.largemessage.store.PayloadStore . | String | None | Yes |
large.message.threshold.bytes | The maximum size of the message is considered large. | Long | 1MB (Default of max.message.bytes) | No |
3.org.apache.kafka.common.
...
serialization.largemessage.LargeMessageDeserializer
Configuration
| Key | Description | Type | Default | Required |
|---|---|---|---|---|
large.message.payload.store.class | Implementation of org.apache.kafka.common.serialization.largemessage.store.PayloadStore . | String | None | Yes |
large.message.skip.not.found.error | Skip not found error when the payload is not found in the store. This allows the deserializer to skip not-found messages and return empty bytes instead new byte[0] . This is important when external store ttl is smaller than kafka retention. | Boolean | FALSE | No |
4.org.apache.kafka.common.
...
serialization.largemessage.PayloadStore
| Code Block | ||
|---|---|---|
| ||
/**
* The contract for any PayloadStore implementation.
* And extract them from the provided configs.
*/
public interface PayloadStore implements Configurable, Closeable, Monitorable {
/**
* Publish data into the store.
*
* @param data data that will be published to the store.
* @return {@link PayloadResponse}.
*/
public abstract PayloadResponse publish(String topic, byte[] data);
/**
* Download full data from the store.
*
* @param path id of the data's reference in the store
* @return content of the object as bytes.
*/
public abstract byte[] download(String path);
/**
* Generate an id for the data's reference in the store.
* By default the id is a random UUID however some stores might need more smarter way to calculate its reference id.
* In such a case please override this method.
* @param data data that will be published to the store.
public String id(byte[] data) {
return UUID.randomUUID().toString();
}
} |
5. org.apache.kafka.common.
...
serialization.largemessage.PayloadResponse
| Code Block | ||
|---|---|---|
| ||
/**
* Response from publish / download from PayloadStore back to the serialization layer
* It contains the final path, response code and the encountered exception if there was any.
* If PlayloadResponse contains PayloadStoreException with isRetryable flag then it will serialiser will
* retry.
*/
public class PayloadResponse {
public final String fullPayloadPath;
public final PayloadStoreException payloadStoreException;
/**
* Construct payload response with response code and payload id.
*/
public PayloadResponse(String fullPayloadPath) {
this(fullPayloadPath, null);
}
/**
* Construct payload response with payload id and exception.
*/
public PayloadResponse(String fullPayloadPath, PayloadStoreException payloadStoreException) {
this.fullPayloadPath = fullPayloadPath;
this.payloadStoreException = payloadStoreException;
}
} |
6. org.apache.kafka.common.
...
serialization.largemessage.PayloadStoreException
| Code Block |
|---|
/**
* Exception class that can either be reliable or not
* this helps the serializer/desrializer to decided either to retry or to crash.
* One subclass will be added is PayloadNotFoundException which is used to indicated if the payload not found
* This is used by deserializer to skip or not.
**/
public class PayloadStoreException extends RuntimeException {
/**
* Constructor PayloadStoreException with message and throwable.
*/
public PayloadStoreException(String message, Throwable t) {
super(message, t);
}
/**
* Constructor PayloadStoreException with message.
*/
public PayloadStoreException(String message) {
super(message);
}
/**
* Constructor PayloadStoreException with throwable.
*/
public PayloadStoreException(Throwable t) {
super(t);
}
}
|
...