Page History

...

This KIP proposes an optional, topic-level feature that provides an opportunity for cost savings on remote storage when the scenario is suitable to use.

Currently, Kafka's tiered storage implementation uploads all non-active local log segments to remote storage immediately, even when they are still within the local retention period. 
This results in redundant storage of the same data in both local and remote tiers.

We can see the one redundancy example in the following picture. (just highlight one segement.)

 Whenthere is no requirement for real-time analytics or immediate consumption based on remote storage. It has the following drawbacks:

...

We can take an AWS S3 billing example (last 3 months) to show the cost saving detail:

 
AWS S3 has two main cost items  items for kakfa usage (normally, Kafka and the bucket are placed in the same region so that we can ignore the network transfer cost).

...

In this billing example, the cost would decrease from about 67K per quarter. BTW: if our topic is local:1 day + remote: 7 days . the saving part will be about 10% （27K）

so it mean that if you set 1 day local + 3-7 days remote. the cost-saving for TimedStorage-ByteHrs will range from 10% to 25%..

Note:  You can also refer to Amazon S3 Cost to know more information.

However, this optimization is offered as a topic level optional configuration rather than the default behavior based on followed scenarios:
(1)  Some users/topics rely on remote storage for real-time analytics and need the latest data to be available as soon as possible  (In fact, it only tries to stay as up-to-date as possible,
      but it still can’t include the latest data because the active segment hasn’t been uploaded yet.).
(2)  Some topics may set a very high ratio for remote-to-local retention time. The cost savings amount will be small, so it is mainly to avoid waste. Users may think it is not worth enabling the feature for the topics
(3)  Kafka admin want to reduce the expansion times for local disk when remote storage down for some time. after all. The already uploaded segments are eligible for deletion from broker when not enable the feature for topic.
      You can check the follow picture to understand the logic: 
      if the remote storage outage for short time:  no matter if  you enable the feature. it don't have difference. 
      if the remote storage outage for a medium-term time:  you will need one extra expansion:  the max size is the your saving cost's part.
      if the remote storage outage for a long time:  no matter if  you enable the feature. you should keep do expansion.

...