Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Storing the raw key and the raw value allows us to send those raw information in the DLQ topic without having to infer the right serializer. All metadata, e.g. Exceptions, StackTrace, topic, partitions and offset would be provided in the record headers by default.

Additionally, the ProcessorContext ProcessingContext would need to be available in each ExceptionHandler. It is currently not available in the ProductionExceptionHandler, thus the handle method will need to be overloaded to provide the context and a default implementation needs to be provided to ensure backward compatibility.

...

  1. Capture the initial message key and values bytes and expose them in the ProcessorContextProcessingContext.
  2. Add a new attribute "deadLetterQueueRecord" in the DeserializationHandlerResponse, ProductionExceptionHandlerResponse and ProcessExceptionHandlerResponse (KIP-1033) enum.
  3. Add the processorContext processingContext attribute in the ProductionExceptionHandlerResponse.handle method and ensure backward compatibility with previous implementations of the interface.
  4. Add a new attribute public static final String DEFAULT_ERRORS_DEADLETTERQUEUE_TOPIC_NAME_CONFIG = "errors.deadletterqueue.topic.name".
  5. Change the existing exception handler to produce a DeadLetterQueue record if the parameter errors.deadletterqueue.topic.name is set.

Default Dead letter queue record

Key

Key of the input message, null if triggered by punctuate

Value

  • If available
and possible
  • , contains the value of the input message
  • If triggered by punctuate, "error during punctuate"
  • If messages exceed Kafka maximum size "message exceeding Kafka maximum record size"

Header: exception

Name of the thrown exception

Header: stacktrace

Stacktrace of the thrown exception 

Header: message

Thrown exception message

Header: topic

Source input topic, null if triggered by punctuate

Header: partition

Source input partition, null if triggered by punctuate

Header: offset

Source input offset, null if triggered by punctuate


Public Interfaces

StreamsConfig.java

Code Block
languagejava
public static final String ERRORS_DEADLETTERQUEUE_TOPIC_NAME_CONFIG = "errors.deadletterqueue.topic.name";

.define(ERRORS_DEADLETTERQUEUE_TOPIC_NAME_CONFIG, // required with no default value
       Type.STRING,
       null, /* default */
       Importance.HIGH,

       ERRORS_DEADLETTERQUEUE_TOPIC_NAME_DOC)

...


ProcessingContext.java

Code Block
languagejava
public interface ProcessorContextProcessingContext {

. . . 

/**
* Return the non-deserialized byte[] of the input message key if the context has been triggered by a message.
*
* <p> If this method is invoked within a {@link Punctuator#punctuate(long)
* punctuation callback}, or while processing a record that was forwarded by a punctuation
* callback, it will return null.
*
* <p> If this method is invoked in a sub-topology due to a repartition, the returned key would be one sent
* to the repartition topic.
*
* @return the raw byte of the key of the source message
*/
byte[] source_raw_key();


/**
* Return the non-deserialized byte[] of the input message value if the context has been triggered by a message.
*
* <p> If this method is invoked within a {@link Punctuator#punctuate(long)
* punctuation callback}, or while processing a record that was forwarded by a punctuation
* callback, it will return null.
*
* <p> If this method is invoked in a sub-topology due to a repartition, the returned value would be one sent
* to the repartition topic.
*
* @return the raw byte of the value of the source message
*/
byte[] source_raw_value();


. . . 

}

...

ProductionExceptionHandler.java

Changes:

  • Adding the ProcessorContext ProcessingContext attribute in the handle and handleSerialization methods, ensure the backward compatibility of previous implementations of this handler by providing default implementation. 
  • Adding the public ProducerRecord<byte[], byte[]> deadLetterQueueRecord; attribute in the ProductionExceptionHandlerResponse
  • Deprecate the previous method

...

Code Block
languagejava
public interface DeserializationExceptionHandler extends Configurable {


   /**
    * Inspect a record and the exception received.
    * <p>
    * Note, that the passed in {@link ProcessorContext} only allows to access metadata like the task ID.
    * However, it cannot be used to emit records via {@link ProcessorContext#forward(Object, Object)};
    * calling {@code forward()} (and some other methods) would result in a runtime exception.
    *
    * @param context processor context
    * @param record record that failed deserialization
    * @param exception the actual exception
    */
   @SuppressWarnings("deprecation") // Old PAPI. Needs to be migrated.
   DeserializationHandlerResponse handle(final ProcessorContextProcessingContext context,
                                         final ConsumerRecord<byte[], byte[]> record,
                                         final Exception exception);


   /**
    * Enumeration that describes the response from the exception handler.
    */
   enum DeserializationHandlerResponse {
       /* continue with processing */
       CONTINUE(0, "CONTINUE"),
       /* fail the processing and stop */
       FAIL(1, "FAIL");


       /** an english description of the api--this is for debugging and can change */
       public final String name;


       /** the permanent and immutable id of an API--this can't change ever */
       public final int id;


       public ProducerRecord<byte[], byte[]> deadLetterQueueRecord;


       DeserializationHandlerResponse(final int id, final String name) {
           this.id = id;
           this.name = name;
       }


       public DeserializationHandlerResponse withDeadLetterQueueRecord(ProducerRecord<byte[], byte[]> deadLetterQueueRecord) {
           this.deadLetterQueueRecord = deadLetterQueueRecord;
           return this;
       }
   }
}

...


ProcessingExceptionHandler

With KIP-1033, a similar behavior would be added to the potential new ProcessExceptionHandlerProcessingExceptionHandler: adding adding a public ProducerRecord<byte[], byte[]> deadLetterQueueRecord; attribute in the ProcessExceptionHandlerResponse ProcessingExceptionHandlerResponse 


Compatibility, Deprecation, and Migration Plan

...

  • Managing DeadLetterQueue directly in the DSL by extending the KStreams interface.
  • Providing no default implementation to build the Dead letter queue record and delegating this task to the user.
  • Only providing exception and metadata information in the default DLQ implementation.
  • Adding a new interface, that could be overload by the user, to build the DLQ record.