Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Add a third option to error handling, which should behave like “continue” when error occurs at Converter or SMT. The infrastructure should send the broken byte message directly to SinkTask.

Schema of the broken bytes should be Optional Byte Array, and the value and the key would be raw bytes that we received from Kafka.

SinkTask is then responsible for handling the unparsed bytes input.

...

There is no compatibility issue. Behavior of Kafka Connect does not change unless user explicitly specify the error handling method to be "continue". 

Rejected Alternatives

As suggested by Chris in the email discussion, we have a rejected alternative plan to this. I quoted the plan below:

Configure your connector to send malformed records to a dead letter queue topic, and configure a separate connector to consume from that dead letter queue topic, use the ByteArrayConverter to deserialize records, and send those records to the destination sink.

Compared with this alternative, the point of this proposal is to save the effort to maintain twice as many connectors as necessary. None