Page History

...

Current state: Under Discussion

Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]

...

A common queue scheduling feature is delayed messages where the message is not supposed to be delivered or consumed right away.   This is usually caused by  The use case is a large influx of messages or other activities happening in the system at the moment, the message producer wants to make sure the messages are being consumed/processed in a little bit later time or the message consumption is being spread over a period of time.  Another common use is message retry handling (e.g. retries in the new Kafka Queue feature), when a message consumer/worker cannot process the message due to some transient failures in external systems, usually the worker wants to unacknowledge the message and retry it later.  The retry is usually more ideal to be scheduled at a later time usually with some exponential backoff time interval.

...

Although the new field deliverAfter is specified on the record/message level, Kafka systems mostly work with the messages in batches for performance reasons.  We will also propose setting a deliverAfter field in RecordBatch level.  The messages in the same batch will satisfy the deliverAfter constraint on the batch level.  For example, for message 1 with deliverAfter set to 10:42 and message 2 with deliverAfter set to 10:44, the batch level deliverAfter will be set to 10:44 so both messages in the batch will be satisfying the constraint.  During message prodding time, the producer will either set the batch level deliverAfter explicitly or implicitly (meaning the producerBatch code will set/adjust the batch level deliverAfter based on the message's deliverAfter inside the batch and in some cases it might decide to split the batch if the message level deliverAfter are too far apart.)

Performance

The enqueue of delayed topics will be a bit slower than the regular topic since we need to build the extra DeliverAfterIndex file.

...