DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
...
Storing the raw key and the raw value allows us to send those raw information in the DLQ topic without having to infer the right serializer. All metadata, e.g. Exceptions, StackTrace, topic, partitions and offset would be provided in the record headers by default.
Additionally, the ProcessorContext ProcessingContext would need to be available in each ExceptionHandler. It is currently not available in the ProductionExceptionHandler, thus the handle method will need to be overloaded to provide the context and a default implementation needs to be provided to ensure backward compatibility.
...
- Capture the initial message key and values bytes and expose them in the ProcessorContextProcessingContext.
- Add a new attribute "deadLetterQueueRecord" in the DeserializationHandlerResponse, ProductionExceptionHandlerResponse and ProcessExceptionHandlerResponse (KIP-1033) enum.
- Add the processorContext processingContext attribute in the ProductionExceptionHandlerResponse.handle method and ensure backward compatibility with previous implementations of the interface.
- Add a new attribute public static final String DEFAULT_ERRORS_DEADLETTERQUEUE_TOPIC_NAME_CONFIG = "errors.deadletterqueue.topic.name".
- Change the existing exception handler to produce a DeadLetterQueue record if the parameter errors.deadletterqueue.topic.name is set.
Default Dead letter queue record
Key | Key of the input message, null if triggered by punctuate |
Value |
|
| |
Header: exception | Name of the thrown exception |
Header: stacktrace | Stacktrace of the thrown exception |
Header: message | Thrown exception message |
Header: topic | Source input topic, null if triggered by punctuate |
Header: partition | Source input partition, null if triggered by punctuate |
Header: offset | Source input offset, null if triggered by punctuate |
Public Interfaces
StreamsConfig.java
| Code Block | ||
|---|---|---|
| ||
public static final String ERRORS_DEADLETTERQUEUE_TOPIC_NAME_CONFIG = "errors.deadletterqueue.topic.name";
.define(ERRORS_DEADLETTERQUEUE_TOPIC_NAME_CONFIG, // required with no default value
Type.STRING,
null, /* default */
Importance.HIGH,
ERRORS_DEADLETTERQUEUE_TOPIC_NAME_DOC) |
...
ProcessingContext.java
| Code Block | ||
|---|---|---|
| ||
public interface ProcessorContextProcessingContext { . . . /** * Return the non-deserialized byte[] of the input message key if the context has been triggered by a message. * * <p> If this method is invoked within a {@link Punctuator#punctuate(long) * punctuation callback}, or while processing a record that was forwarded by a punctuation * callback, it will return null. * * <p> If this method is invoked in a sub-topology due to a repartition, the returned key would be one sent * to the repartition topic. * * @return the raw byte of the key of the source message */ byte[] source_raw_key(); /** * Return the non-deserialized byte[] of the input message value if the context has been triggered by a message. * * <p> If this method is invoked within a {@link Punctuator#punctuate(long) * punctuation callback}, or while processing a record that was forwarded by a punctuation * callback, it will return null. * * <p> If this method is invoked in a sub-topology due to a repartition, the returned value would be one sent * to the repartition topic. * * @return the raw byte of the value of the source message */ byte[] source_raw_value(); . . . } |
...
ProductionExceptionHandler.java
Changes:
- Adding the ProcessorContext ProcessingContext attribute in the handle and handleSerialization methods, ensure the backward compatibility of previous implementations of this handler by providing default implementation.
- Adding the public ProducerRecord<byte[], byte[]> deadLetterQueueRecord; attribute in the ProductionExceptionHandlerResponse
- Deprecate the previous method
...
| Code Block | ||
|---|---|---|
| ||
public interface DeserializationExceptionHandler extends Configurable {
/**
* Inspect a record and the exception received.
* <p>
* Note, that the passed in {@link ProcessorContext} only allows to access metadata like the task ID.
* However, it cannot be used to emit records via {@link ProcessorContext#forward(Object, Object)};
* calling {@code forward()} (and some other methods) would result in a runtime exception.
*
* @param context processor context
* @param record record that failed deserialization
* @param exception the actual exception
*/
@SuppressWarnings("deprecation") // Old PAPI. Needs to be migrated.
DeserializationHandlerResponse handle(final ProcessorContextProcessingContext context,
final ConsumerRecord<byte[], byte[]> record,
final Exception exception);
/**
* Enumeration that describes the response from the exception handler.
*/
enum DeserializationHandlerResponse {
/* continue with processing */
CONTINUE(0, "CONTINUE"),
/* fail the processing and stop */
FAIL(1, "FAIL");
/** an english description of the api--this is for debugging and can change */
public final String name;
/** the permanent and immutable id of an API--this can't change ever */
public final int id;
public ProducerRecord<byte[], byte[]> deadLetterQueueRecord;
DeserializationHandlerResponse(final int id, final String name) {
this.id = id;
this.name = name;
}
public DeserializationHandlerResponse withDeadLetterQueueRecord(ProducerRecord<byte[], byte[]> deadLetterQueueRecord) {
this.deadLetterQueueRecord = deadLetterQueueRecord;
return this;
}
}
}
|
...
ProcessingExceptionHandler
With KIP-1033, a similar behavior would be added to the potential new ProcessExceptionHandlerProcessingExceptionHandler: adding adding a public ProducerRecord<byte[], byte[]> deadLetterQueueRecord; attribute in the ProcessExceptionHandlerResponse ProcessingExceptionHandlerResponse
Compatibility, Deprecation, and Migration Plan
...
- Managing DeadLetterQueue directly in the DSL by extending the KStreams interface.
- Providing no default implementation to build the Dead letter queue record and delegating this task to the user.
- Only providing exception and metadata information in the default DLQ implementation.
- Adding a new interface, that could be overload by the user, to build the DLQ record.