...

People use Kafka, among many reasons, because they need to be sure their messages are correctly processed by their applications. Classic configuration is to have 3 replica, and commit the offset of a message once it has been correctly processed. Developers use this configuraton configuration because it is important not to lose any messages. More important that not losing messages is to be notified when a message is lost without being consumed.

There Nevertheless, there are some situations where messages are lost silently:

• Message expires before being consumed due to topic retention time.
• Message expires before being consumed due to topic size limit.

I propose to build a mechanism to log a warning when after a message is going to/ has been removed due to topic time/size retention settings, for a set of consumer groups especified specified on the topic configuration.

...

• offset of the message that will be has been removed.
• last offset consumed from a consumer group.

Public Interfaces

The kafka-topic.sh tool must understand a new property on --config property:

A new topic level property would be created:

• retention.notify.groups notify.groups.on.expiration : comma separated list of groups that will be notified on offset expiration.

...

The modifications introduced are in blue on the following list:

• The scheduler LogManager is triggered
• The scheduler LogManager will search for the logs to be deleted.
• Read The LogManager read the last offset consumed by all groups specified on "retention.notify.groups.on.expiration" property.
• The scheduler LogManager will remove the log.
• If the offset that has been removed is lower that bigger than the last consumed offset for each group, log a line:
  • "message with offset %d partition %d topic %s key %s has been removed without being consumed by group %s"

...