...

KIP-405 introduced the Tiered Storage feature, which allows to scale the cluster storage independently from CPU and memory and, at the same time, improve , also improving maintenance operations. For each leader's partition log, there is a periodic RLMCopyTask that copies inactive and locally expired segments to the configured remote storage system, along with their auxiliary state (indexes, producer state snapshots, leader-epoch checkpoints). Each remote segment copy is associated with a metadata object , that contains a UUID for that operation. If the same segment copy is retried, a new metadata object with a new UUID is createdspecific copy operation. By default, these metadata are store in an internal non-compacted topic called __remote_log_metadata, and they include the remote segment state, which is initialized to COPY_SEGMENT_STARTED.

When a segment copy fails with a RemoteStorageException, we immediately try to delete the remote segment data, and the operation will be automatically retried in the next task iterationthen a new copy operation is triggered by the next RLMCopyTask execution. If there is a persistent network or remote storage or network issue, the cleanup operation can also fail, but it is never retried. This leaves the segment in a dangling state, because it never transitions to COPY_SEGMENT_FINISHED. Dangling segments are not counted in the periodic retention size calculation, but their uploaded data accumulate in the remote storage if not manually deleted. This may cause storage exhaustion, or additional expense in case of cloud storage.issue, both copy and delete operations fail. The problem is that, while a new copy operation is triggered by the next RLMCopyTask execution and eventually succeed, the delete operation is never retried once it fails. This leaves a certain amount of invalid segment data in the remote storage (wasted space). The consequence of this could be remote storage exhaustion, or unnecessary cloud storage expense. Additionally, the current logic is complicated and fragile, because the COPY_SEGMENT_STARTED actually represents more than one segment state (copy is ongoing and copy failed).

Dangling state

Public Interfaces

The RemoteLogSegmentState public enumeration is declared evolving, and it will be changed to add the DANGLING state.

@InterfaceStability.Evolving
public enum RemoteLogSegmentState {

    /**
     * This state indicates that the segment copy and/or deletion failed possibly leaving some data in the remote storage.
     */
    DANGLING((byte) 4);

Proposed Changes

We propose to introduce a new non-terminal remote segment state called DANGLING, which would be set if when the segment copy fails. At the end of RLMCopyTask.copyLogSegment iteration, the new state will be used to filter all dangling segments and attempt a cleanup deletion transitioning to fail (no deletion will be attempted at this stage). After copying all eligible segments, the RLMCopyTask will try to delete each dangling segment, moving the state to DELETE_SEGMENT_STARTED state.  If the deletion attempt delete operation fails, the state will transition move back to DANGLING, so that we can retry in the next iteration. After this change, RLMCopyTask execution, otherwise the remote segment state will clearly reflect the actual segment's state, and it won't count against the retention size calculation computed in RLMExpirationTask.cleanupExpiredRemoteLogSegments.This is the updated state transition diagram:will move to DELETE_SEGMENT_FINISHED, which is a terminal state.

+--------------------+         +---------------------+
|COPY_SEGMENT_STARTED|-------->|COPY_SEGMENT_FINISHED|
+--+--------------+--+         +--+------------------+
   |              |               |
   |              |               |
   v              v               v
+--+-+----+     +---+---------------+---+
|DANGLING|---->|DELETE_SEGMENT_STARTED|
+---+----+     +--+--------+----------+
    ^             |        |
    |_____________|        |
                           v
               +-----------+-----------+
               |DELETE_SEGMENT_FINISHED|
               +-----------------------+

Public Interfaces

The RemoteLogSegmentState public enumeration is declared evolving, and it will be changed to add the DANGLING state.

@InterfaceStability.Evolving
public enum RemoteLogSegmentState {

    /**
     * This state indicates that the segment copy failed and the broker will attempt to clean any uploaded data.
     * A segment copy could fail because of a network or remote storage issue.
     */
    DANGLING((byte) 4);

...

We will set the new DANGLING state in RLMCopyTask.copyLogSegment, just before attempting to delete the remote segment that failed the copy operation (catch block).

Right after RLMCopyTask.copyLogSegment we will call a new method called cleanupDanglingRemoteLogSegments to filter all dangling remote segments for that partition using the new DANGLING state, and then attempt a deletion.

New broker metrics will be added to track the current number and total size of all dangling remote segments, while related metrics will be updated.

...

Compatibility, Deprecation, and Migration Plan

...

Existing unit tests will be updated as needed, and a new unit test created to verify the dangling remote segment cleanupnew behavior.

Rejected Alternatives

There is no rejected alternative.