THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
| Use cases | Option 1 (Message contains CreateTime + LogAppendTime) | option 2 (Message contains LogAppendTime only) | option 3 (message contains CreateTime only, brokers keep LogAppendTime in log index) | Comparison |
|---|---|---|---|---|
| Mirror Maker | Broker overrides the LAT and keep the CT as is. | Broker overrides the LAT | Broker keep the CT as. And add index entry with LAT to the log index file. | Option 1 provides the most information to user. The only concern is whether we should expose LAT to user. Option 2 loses the CreateTime information. Option 3 have same amount information as option 1 from broker point of view. From user point of view, it does not expose the LAT. |
| Log Retention | Broker will use the LAT of the last message in a segment to enforce the policy. | Same as option 1. | Broker will use the LAT of the last entry in the log index file to enforce the retention policy. Because the leader is the source of truth for LAT, followers need to get the LAT from leader when they replicate the messages. That means we need to introduce a new wire protocol to fetch the time based log index file as well. When log recovery happens, the rebuilt time index would have different LAT from the actual arrival time of the messages in the log. And the LAT in the index file will be very close, or even the same. | Option 1 and option 2 can work with existing replication design and solve the log retention issue we have now. Option 3 alone can not solve the problem we have now. We need additional replication protocol to solve the log retention problemsolves the issue but is a bit involved because it needs to replicate the time index entry as well. |
| Log rolling | Broker will use the LAT of the first message in a log segment to enforce the policy. | Same as option 1. | Broker will use the LAT of the first entry in the log index file to enforce the retention policy. Similar to the log retention case, the followers needs to replicate the time index as well. The log recovery happens, the log rolling might not be honored either. | Option 1 and option 2 solves the log rolling issue. Option 3 solves the issue but is a bit involved because it needs to replicate the time index entry as well. |
| Stream processing | Applications don't need to include the CreateTime in the payload but simply use the CreateTime field. | Applications have to put CreateTime into the payload. | Applications don't need to include the CreateTime in the payload but simply use the CreateTime field. | The benefit of having a CreateTime with each message rather than put it into payload is that application protocol can be simplified. It is convenient for the infrastructure to provide the timestamp so there is no need for each application to worry about the timestamp. |
| Latency measurement | User can get End2End latency and lag in time. | User can get the lag in time. | User can get End2End latency. | Option 1 has most information for user. |
| Search message by timestamp. | Detail discussion in KIP-33 | Detail discussion in KIP-33 | Detail discussion in KIP-33 | Detail discussion in KIP-33 |
...