DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
...
We can see the one redundancy example in the following picture. (I just highlight one segementsegment.)
Whenthere is no requirement for real-time analytics based on remote storage directly. It has the following drawbacks:
1. Wastes storage capacity and costs: The same data is stored twice during the local retention window
2. Provides no No immediate benefit: During the local retention period, reads prioritize local prioritize local data, making the remote copy unnecessary
...
We can take an AWS S3 billing example (last 3 months) to show the cost saving detail:
AWS S3 has two main cost items for kakfa Kafka usage (normally, Kafka deployed machines and the remote storage/S3 bucket are placed in the same region so that we can ignore the network transfer cost).
...
In this billing example, the cost would decrease about 67K per quarter. BTW: if our topic is local:1 day + remote: 7 days . the saving part will be about 10% (27K)
so it mean that if you set 1 day local + 3-7 days remote. the cost-saving for TimedStorage-ByteHrs will range from 25% to 10%..
Note: You can also refer to Amazon S3 Cost to know more information.
However, this optimization is offered as a topic level optional configuration rather than the default behavior based on followed scenarios:
(1) Some users/topics rely on remote storage for real-time analytics based on remote storage directly and need the latest data to be available as soon as possible (In fact, it only tries to stay as up-to-date as possible,
but because it still can’t include the latest data because the active segment always hasn’t been uploaded yet.).
(2) Some topics may set a very high ratio for remote-to-local retention time. The cost savings amount will be small, so it is mainly to avoid waste. Users may think it is not worth enabling the feature for the topics
(3) Kafka admin want to reduce the expansion times for local disk when remote storage down for some a long time. after all. The already uploaded segments are eligible for deletion from broker when not enable the feature for topic.
You can check the follow picture to understand the logic:
if the remote storage outage for a short time: no matter if if you enable the feature. it don't have difference.
if the remote storage outage for a medium-term time: you will need one extra expansion: the max size is the your saving cost's part.
if the remote storage outage for a long time: no matter if you enable the feature. you should keep do doing expansion.
Public Interfaces
This KIP introduces one new topic configuration item: remote.log.latest.enable
BTW: topic's remote storage feature already had some others configure items such as remote.log.delete.on.disable/, remote.log.copy.disable, etc.
...
The storage compare in my test: P2 Partition 2 (host on the machine with the delay upload feature) is about 50% of P0 Partition 0 or P2Partition 1.
BTW: Here are some additional thoughts/considerations.
...




