THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
Kafka connectors which interact with database specially sink connectors need to know how to handle field length mismatches. Most databases like oracle enforce field lengths but there is no way to enforce the same on Avro.
Sample applies with not null fields where records coming in with empty strings need to be rejected.
We probably can write KSQL or stream jobs to filter out these records but the KSQL query can get very big and difficult to manage as there might be hundreds of fields in a table.
An easier approach probably will be in the connect put method to filter out these records using the database metadata already available to the connector and then either discard these bad records or write them into a DLQ topic.
...
This KIP is probably an extension of KIP-298. Particularly the DLQ part which will be used by the connector to send bad records.
These are the DLQ related API changes mentioned in KIP-298.
Config Option | Description | Default Value | Domain |
|---|---|---|---|
| errors.deadletterqueue.context.headers.enable | If true, multiple headers will be added to annotate the record with the error context | false | Boolean |
| errors.deadletterqueue.topic.name | The name of the dead letter queue topic. If not set, this feature will be disabled. | "" | A valid Kafka topic name |
| errors.deadletterqueue.topic.replication.factor | Replication factor used to create the dead letter queue topic when it doesn't already exist. | 3 | [1 ... Short.MAX_VALUE] |
Proposed Changes
The put method inside the connector task should support passing the DLQ producer to it.
...