THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
Option discussions with use cases
This section discusses how the proposed and rejected option works with a few use cases.
Mirror maker
The behavior of broker for all the options are the same: The broker will always override the LogAppendTime(if exists) when message arrives the broker and keep the CreateTime(if exists) untouched.
The broker does not distinguish mirror maker from other producers. The following example explains what will the timestamp look like when there is mirror maker in the picture.(CT - CreateTime, LAT - LogAppendTime)
...
The LogAppendTime of a message in source cluster and target cluster will be different.
Comparison
Proposed Option (CreateTime + LogAppendTime) | rejected option 1 (LogAppendTime only) | rejected option 2 (CreateTime + LogAppendTime in log index) | Comparison | |
|---|---|---|---|---|
| Mirror Maker | Broker overrides the LAT and keep the CT as is. | Broker overrides the LAT | Broker keep the CT as. And add index entry with LAT to the log index file. | Proposed option provides the most information to user. The only concern is whether we should expose LAT to user. Rejected option 1 loses the CreateTime information. Rejected option 2 have same amount information as proposed option from broker point of view. From user point of view, it does not expose the LAT. |
| Log Retention | Broker will use the LAT of the last message in a segment to enforce the policy. | Same as proposed option. | Broker will use the LAT of the last entry in the log index file to enforce the retention policy. Because the leader is the source of truth for LAT, followers need to get the LAT from leader when they replicate the messages. That means we need to introduce a new wire protocol to fetch the time based log index file as well. When log recovery happens, the rebuilt time index would have different LAT from the actual arrival time of the messages in the log. And the LAT in the index file will be very close, or even the same. | Proposed option and rejected option 1 can work with existing replication design and solve the log retention issue we have now. rejected option 2 alone can not solve the problem we have now. We need additional replication protocol to solve the log retention problem. |
| Log rolling | Broker will use the LAT of the first message in a log segment to enforce the policy. | Same as proposed option. | Broker will use the LAT of the first entry in the log index file to enforce the retention policy. Similar to the log retention case, the followers needs to replicate the time index as well. The log recovery happens, the log rolling might not be honored either. | Proposed option and rejected option 1 solves the log rolling issue. Rejected option 2 does not solve the problem and needs additional replication protocol. |
| Stream processing | ||||
| Latency measurement |
Broker side use cases
To discuss the usage of CreateTime and LogAppendTime, it is useful to summarize the latency pattern of the messages flowing through the pipeline. The latency can be summarized to the following pattern:
...