THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
This section discusses how the proposed and rejected option works with a few use cases.
Comparison
Options comparison
| Use cases | Proposed Option (Message contains CreateTime + LogAppendTime) | rejected option 1 (Message contains LogAppendTime only) | rejected option 2 (message contains CreateTime only, brokers keep LogAppendTime in log index) | Comparison |
|---|---|---|---|---|
| Mirror Maker | Broker overrides the LAT and keep the CT as is. | Broker overrides the LAT | Broker keep the CT as. And add index entry with LAT to the log index file. | Proposed option provides the most information to user. The only concern is whether we should expose LAT to user. Rejected option 1 loses the CreateTime information. Rejected option 2 have same amount information as proposed option from broker point of view. From user point of view, it does not expose the LAT. |
| Log Retention | Broker will use the LAT of the last message in a segment to enforce the policy. | Same as proposed option. | Broker will use the LAT of the last entry in the log index file to enforce the retention policy. Because the leader is the source of truth for LAT, followers need to get the LAT from leader when they replicate the messages. That means we need to introduce a new wire protocol to fetch the time based log index file as well. When log recovery happens, the rebuilt time index would have different LAT from the actual arrival time of the messages in the log. And the LAT in the index file will be very close, or even the same. | Proposed option and rejected option 1 can work with existing replication design and solve the log retention issue we have now. rejected option 2 alone can not solve the problem we have now. We need additional replication protocol to solve the log retention problem. |
| Log rolling | Broker will use the LAT of the first message in a log segment to enforce the policy. | Same as proposed option. | Broker will use the LAT of the first entry in the log index file to enforce the retention policy. Similar to the log retention case, the followers needs to replicate the time index as well. The log recovery happens, the log rolling might not be honored either. | Proposed option and rejected option 1 solves the log rolling issue. Rejected option 2 does not solve the problem and needs additional replication protocol. |
| Stream processing | Applications don't need to include the CreateTime in the payload but simply use the CreateTime field. | Applications have to put CreateTime into the payload. | Applications don't need to include the CreateTime in the payload but simply use the CreateTime field. | The benefit of having a CreateTime with each message rather than put it into payload is that application protocol can be simplified. It is convenient for the infrastructure to provide the timestamp so there is no need for each application to worry about the timestamp. |
| Latency measurement | User can get End2End latency and lag in time. | User can get the lag in time. | User can get End2End latency. | Proposed option has most information for user. |
| Search message by timestamp. | Detail discussion in KIP-33 | Detail discussion in KIP-33 | Detail discussion in KIP-33 | Detail discussion in KIP-33 |
Mirror maker case in detail
...
The behavior of broker for all the options are the same: The broker will always override the LogAppendTime(if exists) when message arrives the broker and keep the CreateTime(if exists) untouched.
...