Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

 Whenthere is no requirement for real-time analytics or immediate consumption from based on remote storage. It has the following drawbacks:

1. Wastes storage capacity and costs: The same data is stored twice during the local retention window
2. Provides no immediate benefit: During the local retention period, reads prioritize local data, making the remote copy unnecessary


Example Scenario and The scenario and this KIP's goal:

Consider a topic with remote storage enable:
- Local retention: 1 day (24 hours)
- Remote retention: 3 days (72 hours)

Currently, data Data is stored in both tiers for the first day, resulting in about 16 hours of redundant storage. This can leads to cost increaseswaste.

So we can reduce the tiered storage redundancy for cost saving. In this case, it will save about 25% cost pay payment for total size of disk.



However, some users/topics rely on remote storage for real-time analytics and need the latest data to be available as soon as possible

 (In fact, it only tries to stay as up-to-date as possible, but it still can’t include the latest data because the active segment hasn’t been uploaded yet.).

Therefore, this optimization is offered as a topic's optional configuration rather than the default behavior.

...

This KIP introduces one new topic configuration item:  remote.log.keep.latestlatest 
BTW:  topic's remote storage already had some others items such as remote.log.delete.on.disable/remote.log.copy.disable, etc.


Code Block
languagejava
titleclients/src/main/java/org/apache/kafka/common/config/TopicConfig.java‎
public static final String REMOTE_LOG_KEEP_LATEST_CONFIG = "remote.log.keep.latest";
public static final String REMOTE_LOG_KEEP_LATEST_DOC = "Determines whether to upload all segments to remote storage including the latest ones within local retention. " +
            "When set to true (default), all committed segments will be uploaded without checking local retention constraints. " +
            "When set to false, only segments beyond local retention period will be uploaded to remote storage.";

...

You can see the uploading will be delayed if the configure remoteLogKeepLatest is false. And After the change, the remote tiered storage redundancy will be reduced with delayed upload.
You can refer to the test case and result: https://github.com/apache/kafka/pull/20913#issuecomment-3547156286

BTW: Here are some additional thoughts/considerations.

  1. Local files won’t be deleted until they’ve been uploaded to the remote storage, so this change is very safe
    You don’t need to worry about files being cleaned up before they be upload to the remote.
  2. Considering the latency of remote storage, the local retention period won’t be set too short.
    For example, in our production environment, we keep one day of local data alongside 3-7 days in remote storage, so there’s still one day of redundancy.

Compatibility, Deprecation, and Migration Plan 

...