...

• org.apache.kafka.common.record
• org.apache.kafka.clients.producer
• org.apache.kafka.streams.processor

Proposed Changes

Add a topic property that specifies that topic may have a valid negative timestamps.

First, we need to remove all checks for negative timestamps across the code:

• client should be able to publish record with a negative timestamp (to the topics that support that),
• broker should accept and serve that record,
• streams should not drop a record with a negative timestamp.

NO_TIMESTAMP (−1) problem

The broker uses −1 as a default value for missing timestamp. Which might be a correct value set by the user: Wednesday, December 31, 1969 11:59:59 PM UTC.

 Prosed Proposed behavior is to ignore change that problem and it is a choice of the user to:

...

semantics and use Long.MIN_VALUE for messages without timestamp for new topics.

Streams will ignore negative record timestamp if a topic was made without negative timestamp support.

We may need a new timestamp type along with CreateTime and LogAppendTime to prevent legacy brokers from writing −1 (with meaning no timestamp) to the new topics.

Compatibility, Deprecation, and Migration Plan

...

If we need special migration tools, describe them here.No need if

we are not adding field "hasTimestamp" to the messageA migration tool is not required. Users may write their own consumer/producer to copy all data from old topic to topic that supports negative timestamp.
It's up to the users how to threat −1 timestamp from the old topic: convert to Long.MIN_VALUE or rely on broker wall clock.

When will we remove the existing behavior?

...

If there are alternative ways of accomplishing the same thing, what were they? The purpose of this section is to motivate why the design is the way it is and not some other way.Topic property:

Add a topic property that says if it may have records with "no timestamp'. That case:

1. users would need to create a new topic and migrate/stream all the records from old topic to the new one
2. users decide what to do with a record without a timestamp: set the timestamp to current or some specific one based on a message content.

Message flag "hasTimestamp":

1. add a special boolean flag to message record "hasTimestamp",
2. write a migration tool that adds this flag to message with the negative timestamp to legacy messages,
3. make sure clients know about that field and check them

Ignore −1 problem:

The solution could be just to ignore that problem and it is a choice of the user to:

1. interpret this value as a real timestamp
2. interpret −1 as "no timestamp" and other values as a real timestamp (can we borrow 1 millisecond for our needs?).

Timestamp delta:

In case it's known that you would need no more than 200 years back before Unix epoch, the possible solution is:

• producers should add that "delta" to an original timestamp,
• all consumers who do lookups by timestamp should know about that "delta",
• keep −1 semantics as it is now.

It may seem the easiest thing to do, nevertheless changing −1 semantics is a cleaner solution.