...

Proposed Changes

Error Reporter Object

...

Kafka Connect The SinkTaskContext class will create a BiConsumer<SinkRecordBiFunction<SinkRecord, Throwable, Throwable>Future<Void>> object to as the error reporter object. Along with the original errant sink record, the exception thrown will be sent to the error reporter to provide additional context. The only configurations exposed to the user and developer will be the following, taken from KIP-298:

...

Config Option

...

Description

...

Default Value

...

Domain

...

For asynchronous functionality, the error reporter will return a Future for each record processed. The error reporter will use the same configurations as the dead letter queue in KIP-298 to avoid redundant configuration. There will be no additional configurations for the Producer

...

and AdminClient under the hood aside from the

...

existing producer.

...

 and admin.

...

 configurations present in the worker configurations

...

and producer.override.

...

 and admin.override.

...

 configurations present in the connector configurations.

...

 Serialization for the errant records will be done in the same manner as KIP-298.

Oconfigured to use an error reporter, the task can send errant records to it within put(...)

Synchrony

The error reporting functionality is intended to be synchronous. If a record is sent to the error reporter, can be either asynchronous or synchronous. By default, error reporting will be asynchronous; processing of the next subsequent errant record in accept(...)  will not be blocked by the successful processing of the prior errant record. However, if the developer prefers synchronous functionality, they can block processing of the next record with future.get(). In order to avoid error records being written out of order (for example, due to retries), the developer can use max.in.flight.requests.per.connection=1 in their implementation for writing error records.

Additionally, begin until the producer successfully sends the errant record to Kafka. If the producer encounters an exception while sending the errant record to Kafka, the task will be killed; if the error reporting is failing as well, it does not make sense to keep running the connector. While this synchronous design will cause a throughput hit if many records are sent to the error reporter, a high number of errant records should not be a typical case in connector usage. Additionally, this throughput hit can provide a chance for the user to realize there is a high rate of errant records before many records are actually sent to the error reporter. 

Metrics

The following metrics will used to monitor the error reporter:

...