Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

So we can reduce the tiered storage redundancy for cost saving. In this case, it will save about 25% cost payment for total size of disk.



We can take an AWS S3 billing example (last 3 months):
Image Modified
 
S3 has two cost items  (normally, Kafka and the bucket are placed in the same region so that we can ignore the network transfer cost).

The cost item “TimedStorage-ByteHrs” represents the storage part. In above case, each remote segment lives for about 64 hours. thus, with the new optional solution, each will live for about 48 hours, resulting in approximately 25% cost savings.

In this billing example, the cost would decrease from 67K per quarter.

Note:  You can also refer to Amazon S3 Cost to know more information.


However, this optimization is offered as a topic's optional configuration rather than the default behavior based on followed scenarios:
(1) 
some users/topics rely on remote storage for real-time analytics and need the latest data to be available as soon as possible  (In fact, it only tries to stay as up-to-date as possible,
      but it still can’t include the latest data because the active segment hasn’t been uploaded yet.).
(2)  user want 



However,

Therefore, this optimization is offered as a topic's optional configuration rather than the default behavior. 

Public Interfaces

This KIP introduces one new topic configuration item:  remote.log.keep.latest 
BTW:  topic's remote storage already had some others items such as remote.log.delete.on.disable/remote.log.copy.disable, etc.

...