DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
| Table of Contents |
|---|
Status
Current state: Draft [One of " Under Discussion", "Accepted", "Rejected"]
Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]
JIRA: here [Change the link from KAFKA-1 to your own ticket]
...
Apache Kafka already has dead-letter queues for Kafka Connect and Kafka Streams. A dead-letter queue in Kafka is a topic but “dead-letter queue” is a term already used in the industry. This KIP extends the idea to apply to share groups as well.
Proposed Changes
In Kafka, a "dead-letter queue" is a topic onto which records whose delivery failed are automatically written. A share group can be configured with the name of a topic to be used as the group's DLQ topic. When an undeliverable record transitions to the Archived state for such a group, a record is written onto the DLQ topic. This DLQ record contains the context of the undeliverable record (topic name, partition, offset and so on) as record headers, and optionally the content of the undeliverable record itself.
The following context headers are included in the DLQ record, overwriting existing headers of the same name if they already exist in the original record. To avoid clashing with headers from the original record, all error context headers start with "__dlq.errors." :
| Header name | Description |
|---|---|
__dlq.errors.topic | The name of the topic that contained the original record. |
__dlq.errors.partition | The partition that contains the original record (encoded as a UTF-8 string). |
__dlq.errors.offset | The offset of the original record (encoded as a UTF-8 string). |
__dlq.errors.group | If known, the group ID of the delivery of the original record. |
__dlq.errors.delivery.count | If known, the delivery count of the original record (encoded as a UTF-8 string). |
__dlq.errors.exception.class.name | If known, the fully-qualified class name of the exception that was thrown during the final delivery attempt. |
__dlq.errors.exception.message | If known, the message of the exception that was thrown during the final delivery attempt. |
The "__dlq.errors.exception." headers are defined for situations in which the cause of the failure could be identified and included in the DLQ record, such as a RecordDeserializationException .
...
Configuration
Broker configuration
| Configuration | Description | Values |
|---|---|---|
errors.deadletterqueue.copy.record.enable | Whether to allow copying the original record onto the dead-letter queue topic for failed record delivery. | Type: boolean, default: true |
errors.deadletterqueue.topic.name.prefix | The prefix of permitted dead-letter queue topic names. If this is set to "", there is no restriction on the names used for dead-letter queue topics. | Type: string, default: "dlq." |
Group configuration
Note that these configuration property names do not begin with share even though they are initially only relevant to the behavior of share groups. This is because the concept is likely to be applicable to other group types in the future.
| Configuration | Description | Values |
|---|---|---|
errors.deadletterqueue.topic.name | The name of the topic to be used as the dead-letter queue (DLQ) for undeliverable records. The topic name is blank by default, which means that no records are to be records in the DLQ. | Type: string, default: "" |
errors.deadletterqueue.copy.record.enable | Whether to copy the record onto the dead-letter queue topic, or just to write a record containing the context information headers. | Type: boolean, default: false |
Compatibility, Deprecation, and Migration Plan
...