Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  1. Local files won’t be deleted until they’ve been uploaded to the remote storage, so this change is very safe
    You don’t need to worry about files being cleaned up before they be upload to the remote.
  2. Considering the latency of remote storage, the local retention period won’t be set too short.
    For example, in our production environment, we keep we keep one day of local data alongside 3-7 days in remote storage,
    Keeping one day of data in local storage ensures that Kafka users have enough time to handle unexpected consumption issues while maintaining good latency, and it also reduces the risk of relying on remote storage.
    so there’s still one day of redundancy.

...