Status

Current state: "Under Discussion"

Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]

JIRA: here

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

 Currently, Kafka Streams does not support ProcessingExceptionHandler for GlobalKTable processors. When a processing exception occurs during GlobalKTable record processing, the GlobalStreamThread fails and terminates, causing the entire Kafka  Streams application to shut-down. This behavior is inconsistent with regular KStream/KTable processing, where ProcessingExceptionHandler allows applications to handle exceptions gracefully and continue processing.

Public Interfaces

No changes in public interfaces. We will be reusing existing interfaces. 

Proposed Changes

This KIP extends the applicability of the existing ProcessingExceptionHandler to GlobalKTable processors. Currently, this exception handler only applies to regular stream processing (KStream/KTable). After this KIP, the same handler will also
handle processing exceptions in GlobalKTable.

Exception Handling Scope

After this KIP, the ProcessingExceptionHandler configured via `default.processing.exception.handler` will be invoked for:

  - KStream processing exceptions (existing)
  - KTable processing exceptions (existing)
  - GlobalKTable processing exceptions (new capability added by this KIP)

 Current State

 DeserializationExceptionHandler is configured via StreamsConfig, instantiated via TopologyConfig, passed to GlobalStateUpdateTask, and used during GlobalKTable deserialization. No Global Processor exception is being handled. We will be replicating the same framework of initialization for ProcessingExceptionHandler in GlobalKTables.

 Proposed Extension

The infrastructure for configuration and instantiation of ProcessingExceptionHandler already exists. However, it is currently only passed to regular stream tasks, not to global state update tasks. This KIP extends the existing pattern to pass ProcessingExceptionHandler to GlobalStateUpdateTask, mirroring how DeserializationExceptionHandler is already passed to both regular and global tasks. This enables exception handling during GlobalKTable processor execution.

Implementation

  Add ProcessingExceptionHandler as a constructor parameter to GlobalStateUpdateTask and pass it to processor initialization, mirroring the existing DeserializationExceptionHandler implementation.

Limitation

In this KIP, DLQ records returned by the handler will be logged but NOT sent to Kafka for GlobalKTable processors. This is because GlobalKTable processing does not currently have producer infrastructure. We will cover DLQ in a separate KIP. 

Compatibility, Deprecation, and Migration Plan

This change is fully backward compatible:
  - Applications without a configured handler experience no behavior change
  - Applications with a configured handler automatically get GlobalKTable support
  - The handler interface is not modified - The handleError() method (introduced in KIP-1034) will be called for GlobalKTable exceptions. 
  - No new configuration properties are introduced

Test Plan

Since this is a fairly small change, unit tests and integration test should be sufficient.

Rejected Alternatives

• Create a new processexceptionalhandler interface for global thread which require introducing new configuration at the stream config level. This would create unnecessary operational  overload for the end users as it require to implement multiple interfaces .