...

People use Kafka, among many reasons, because they need to be sure their messages are correctly processed by their applications. Classic One typical configuration is to have 3 replica, and commit the offset of a message once after it has been correctly processed. Developers use this configuration because it is important not to lose any messages. More But even more important that not losing messages is to be notified when a message is lost without being consumed.

...

I propose to log a warning after a message has been removed due to topic time/size retention settings, for a set of consumer groups specified on the topic configuration.

The I believe this could be implemented because the kafka brokers know the information needed to achieve for the goaltask:

• offset of the message that has been removed.
• last offset consumed from a consumer group.

Public Interfaces

A new property at topic level property would should be created:

• retention.notify.groups : comma separated list of groups that will be notified on offset expiration.

...

Currently the LogManager schedule the "kafka-delete-logs" thread, that will call the deleteLogs() method. Is possible to add into that method a a check to list the consumed groups that didnt didn't consume the offset messages being removed.

The pseudo code is in the comment in the middle below:

...