Status

Current state:  Discussion

...

PR: https://github.com/apache/kafka/pull/20913

Motivation

Currently, Kafka's tiered storage implementation uploads all non-active local log segments to remote storage immediately, even when they are still within the local retention period. 
This results in redundant storage of the same data in both local and remote tiers.

...

So we can reduce the tiered storage redundancy for cost saving. In this case, it will save about 25% cost payment for total size of disk.

Image AddedImage Removed

We can take an AWS S3 billing example (last 3 months) to show the cost saving detail:
Image RemovedImage Added
 
AWS S3 has two cost items  (normally, Kafka and the bucket are placed in the same region so that we can ignore the network transfer cost).

...

In this billing example, the cost would decrease from 67K per quarter.
Note:  You can also refer to Amazon S3 Cost to know more information.

However, this optimization is offered as a topic level optional configuration rather than the default behavior based on followed scenarios:
(1)  Some users/topics rely on remote storage for real-time analytics and need the latest data to be available as soon as possible  (In fact, it only tries to stay as up-to-date as possible,
      but it still can’t include the latest data because the active segment hasn’t been uploaded yet.).
(2)  Some topics may set a very high ratio for remote-to-local retention time. The cost savings amount will be small, so it is mainly to avoid waste. Users may think it is not worth enabling the feature for the topics
(3)  Kafka admin want to reduce the expansion times for local disk when remote storage down for some time. after all. The already uploaded segments are eligible for deletion from broker when not enable the feature for topic.
      You can check the follow picture to understand the logic: 
      if the remote storage outage for short time:  no matter if  you enable the feature. it don't have difference. 
      if the remote storage outage for a medium-term time:  you will need one extra expansion:  the max size is the your saving cost's part.
      if the remote storage outage for a long time:  no matter if  you enable the feature. you should keep do expansion.
Image RemovedImage RemovedImage AddedImage Added

Public Interfaces

This KIP introduces one new topic configuration item:  remote.log.keep.latest 
BTW:  topic's remote storage already had some others items such as remote.log.delete.on.disable/remote.log.copy.disable, etc.

...

The default value is true so that the whole remote storage module keeps the original behavior when topic don't set it to false.

Proposed Changes

You can refer to https://github.com/apache/kafka/pull/20913 for the detailed changes.

We change the RemoteLogManager.RLMCopyTask#candidateLogSegments's logic for decide one segment if need to upload to remote: 

Image RemovedImage Added

You can see the uploading will be delayed if the configure remoteLogKeepLatest is false. And After the change, the remote tiered storage redundancy will be reduced with delayed upload.
You can refer to the test case and result: https://github.com/apache/kafka/pull/20913#issuecomment-3547156286

...

1. Local files won’t be deleted until they’ve been uploaded to the remote storage, so this change is very safe
   You don’t need to worry about files being cleaned up before they be upload to the remote.
2. Considering the latency of remote storage, the local retention period won’t be set too short.
   For example, in our production environment, we keep one day of local data alongside 3-7 days in remote storage, so there’s still one day of redundancy.

Compatibility, Deprecation, and Migration Plan 

• Backward Compatibility
  This change is fully compatible:
  Backward compatible: Default value (true) maintains current behavior
  Forward compatible: Older clients unaware of this config will ignore it
  
  

• Deprecation
  N/A
• Migration for Existing Deployments
  N/A. The feature is optional with topic level.

Test Plan

We can use follow tests to cover the change:

...

1. Verify segments are uploaded before local segment deleted
2. Test the remote storage reduced after topic enable the feature.

Rejected Alternatives

Alternative 1: Make this the default behavior

...