DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
...
- Local files won’t be deleted until they’ve been uploaded to the remote storage, so this change is very safe
You don’t need to worry about files being cleaned up before they be upload to the remote. When local and complete retention values are set to the same. What is the behaviour?
This is very interest corner case (Maybe few person do this thing, but it is interesting case).Actually, this is a valid case because local retention only not allow to > remote retention in current code. If they are equal, it is better to skip the update, as you mentioned, since the segment would be immediately deleted after being uploaded to remote storage.
However, if we do not upload it to remote storage, the local segment will not be deleted because it waits for the highest offset in remote storage to be updated after the upload.
Moreover, if we skip the upload but directly update the highest offset in remote storage, it becomes ambiguous whether the segment has already been uploaded or not.
Therefore, We can skip the upload and update the LogStartOffset. The demo PR is:corner case. So keep same as the existing behaviour; the segment will be uploaded to remote, then allowed for local-log deletion.
Code Block language plain title Note Note: Actually, this is a valid case because local retention only not allow to > remote retention in current code. If they are equal, it is better to skip the update, since the segment would be immediately deleted after being uploaded to remote storage. However, if we do not upload it to remote storage, the local segment will not be deleted because it waits for the highest offset in remote storage to be updated after the upload. Moreover, if we skip the upload but directly update the highest offset in remote storage, it becomes ambiguous whether the segment has already been uploaded or not. Therefore, We can skip the upload and update the LogStartOffset. The demo PR is: https://github.com/apache/kafka/pull/21361
question, Considering this is a corner case and this solution also helps address another issue: if the remote storage service is unavailable for a long time, local segments may never get deleted forever even it over the retention time. It means that it isn't special case for this KIP. So I just list this
for itthought here
.
Compatibility, Deprecation, and Migration Plan
...