...

When a record is repeatedly delivered to a share group without being successfully acknowledged, it may be because the record has some kind of serialisation or semantic problem and is a so-called "poison message". The delivery attempt limit is enforced to prevent an endless loop of failed processing attempts, even in cases where the consuming application does not have adequate error handling and simply fails on each attempt. A well written consuming application can capture a failing record, and explicitly reject it, thus ending its delivery attempts more efficiently. These undeliverable records are candidates for placing on a dead-letter queue (DLQ) topic for inspection or special processing, as opposed to simply being put directly into the the Archived state as described in KIP-932.

...

In Kafka, a "dead-letter queue" is a topic onto which records whose delivery failed are automatically written. A share group can be configured with the name of a topic to be used as the group's DLQ topic. When an undeliverable record For a share group with a DLQ topic, when a record's delivery is rejected by the consumer or the record reaches its maximum number of delivery attempts, it transitions to the Archived Archiving state for such a group, a record is written onto the DLQ topic, and then the record transitions into the Archived state. This DLQ record contains the context of the undeliverable record (topic name, partition, offset and so on) as record headers, and optionally the content of the undeliverable record itself.

The following context information headers are included in the DLQ record, overwriting existing headers of the same name if they already exist in the original record. To avoid clashing with headers from the original record, all error context information headers start with "__dlq.errors." :

...

A broker configuration errors.deadletterqueue.topic.name.prefix  contains the prefix of permitted DLQ topic names. By default, this is "dlq."  meaning that DLQ topics must start with this prefix, such as "dlq.T1" . This instantly tells consuming applications that the records they see were indirectly written to this topic. The context information headers also indicate that these are potentially DLQ records.

...

In addition, the DLQ records do not need to contain the actual data from the original records. The copying of the original record data into the DLQ record is controlled by two configurations, one for the broker and one for the group, with the broker configuration taking precedence. The copying of the record data only occurs if both the broker configuration and the group configuration are enabled. By default, the broker configuration is enabled, while the group configuration is disableda group configuration errors.deadletterqueue.copy.record.enable  which is false by default.

DLQ topic conventions

There are a couple of obvious conventions for the use of DLQ topics.

...

This KIP defines a new DeliveryState  of Archiving  with the value of 3 which indicates a record which is in the process of being written to the dead-letter queue topic. As the details of the implementation develop, it will become clear whether this value is usedwritten to the share-group state topic, or just a state in memory like Acquired. The KIP reserves it for potential use in the existing share coordinator record schemas.

...

The following new broker metrics are added:

Compatibility, Deprecation, and Migration Plan

...