Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Public Interfaces

This KIP introduces:

(1) one new topic configuration item:  remote.logcopy.latestlazy.enable
BTW:  topic's remote storage feature already had some others configure items such as remote.log.delete.on.disable, remote.log.copy.disable, etc.

...

Code Block
languagejava
titleclients/src/main/java/org/apache/kafka/common/config/TopicConfig.java‎
    public static final String REMOTE_LOGCOPY_LATESTLAZY_ENABLE_CONFIG = "remote.logcopy.latestlazy.enable";
    public static final String REMOTE_LOGCOPY_LATESTLAZY_ENABLE_DOC = "Determines whether to upload all non-activedelay uploading segments to remote storage, including those still within local retention. " +
            "When set to truefalse (default), all non-active segments will be uploaded immediately without checking local retention constraints. " +
            "When set to falsetrue, only segments beyondwithin local retention periodwill willnot be uploaded to remote storage.", the upload is delayed until they expire.";

(2) one new server configuration item:  log.remote.copy.lazy.enable

Code Block
languagejava
titleorg.apache.kafka.server.config.ServerLogConfigs
    public static final String LOG_REMOTE_COPY_LAZY_ENABLE_CONFIG = LOG_PREFIX + TopicConfig.REMOTE_COPY_LAZY_ENABLE_CONFIG;
    public static final String LOG_REMOTE_COPY_LAZY_ENABLE_DOC = TopicConfig.REMOTE_COPY_LAZY_ENABLE_DOC;


The default value is true so that the whole remote storage module keeps the original behavior when topic don't set it to false.

...

You can see the uploading will be delayed if the configure remoteLogKeepLatestremote.copy.lazy.enable is falsetrue. And After the change, the remote tiered storage redundancy will be reduced with delayed upload.
You can refer to the test case and result: https://github.com/apache/kafka/pull/20913#issuecomment-3547156286

...