THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
Discussion thread: here
JIRA: here
The proposal discussed in this KIP is implemented in this pull request.
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
...
| Type | Name | Description |
|---|---|---|
| Record | record | The record which caused the exception (if available). |
| Stage[] | stages | An ordered array of stages. |
| int | index | The index into the above array to the stage A pointer to a stage in the array where the failure occurred. |
| string | exception | The exception, and stacktrace for the failure (if available). |
| int32 | attempt | Number of attempts made to correct the failure. |
| string | task_id | The id of the task which where the failure occurred. |
| int64 | time_of_error | The epoch time of failure. |
where Record contains the following fields
| Type | Name | Description | |
|---|---|---|---|
| string | topic | the topic which this message was read from or will be written to. | |
| int64 | timestamp | the timestamp of the Kafka message. | |
| string | timestamp_type | the type of the timestamp. One of: NO_TIMESTAMP_TYPE, CREATE_TIME or LOG_APPEND_TIME. | |
| int32 | partition | the partition which this message was read from or will be written to.SchemaAndValue | |
| int64 | keytimestamp | the | keytimestamp of the Kafka message. |
| SchemaAndValue | key | the key of the message. | |
| SchemaAndValue | value | the value of the message. | |
| SchemaAndValue | headers | the headers of the message. | |
| map<string, string> | source_partition | information about the source partition from where this message was created from (usually, specific to a source connector implementation). | |
| map<string, string> | source_offset | information about the source offset from where this message was created from (usually, specific to a source connector implementation). |
and SchemaAndValue is:
| int64 | offset | the offset of the message in the Kafka topic (available only in sink connectors). |
| string | timestamp_type | the type of the timestamp. One of: NO_TIMESTAMP_TYPE, CREATE_TIME or LOG_APPEND_TIME (available only in sink connectors). |
| map<string, string> | source_partition | information about the source partition from where this message was created from (available only in source connectors and contents are specific to the implementation). |
| map<string, string> | source_offset | information about the source offset from where this message was created from (available only in source connectors and contents are specific to the implementation). |
and SchemaAndValue is:
| Type | Name | Type | Name | Description |
|---|---|---|---|---|
| string | schema | the data type of the value. For example, "BYTES", or "STRUCT" | ||
| string | value | a string encoding of the value. If the type is "BYTES", then this value must be base64 decoded to obtain the original byte[]. If not, it is a JSON representation of the Connect record. |
Finally, the Stage object describes the different operations performed in a Connector pipeline:
| Type | Name | Description |
|---|---|---|
| string | type | the operation type. For example, TRANSFORMATION, KEY_CONVERTER or TASK_PULL. |
| map<string, string> | config | the properties used to configure this object. |
| string | class | the classname of the object performing this operation. |
Metrics
The following new metrics will monitor the number of failures, and the behavior of the response handler. Specifically, the following set of counters:
...