Status

Current state: Under Discussion

...

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

...

Kafka Connect currently supports

...

- `org.apache.kafka.connect.runtime.errors.DeadLetterQueueReporter` - Reports records into a dead letter queue.

- `org.apache.kafka.connect.runtime.errors.LogReporter` - An abstract class for reporting errors via logging. Both sink and source connectors have their own implementations of the log reporter.

...

basic error tolerance mechanisms (`none`, `all`) and dead letter queues (DLQ) for error reporting. However, there is no flexible way for developers to define how errors should be handled during specific stages of record processing.

Kafka Streams already provides this flexibility via the ProcessingExceptionHandler interface, which allows stream applications to determine whether to continue, fail, or log on processing errors. This KIP proposes a similar mechanism for Connect: a pluggable `ErrorHandler` interface for custom error handling logic.

Public Interfaces

We propose the following new public interface:

...

errors.reporters=example

errors.reporters.example.type=com.example.SimpleErrorRecordReporter

errors.reporters.example.param=testValue

Proposed Changes

We propose a new pluggable interface (`ErrorRecordReporter`) which enables the creation of custom error record reporters. These custom reporters will be executed when `errors.tolerance=all`. The draft pull request: https://github.com/apache/kafka/pull/17493

Compatibility, Deprecation, and Migration Plan

There are no backward compatibility concerns.

Rejected Alternatives

1. Exposing `ErrorReporter` and `ProcessingContext` as public APIs: The main issue with this design is that it exposes packages with `runtime` in their names, which cannot be changed.
2. Naming as ErrorReporter: This name is already used for error reporters in the runtime module. To avoid name conflicts, `ErrorRecordReporter` was chosen.