Status

Current state: Under Discussion

Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]

JIRA:  KAFKA-17428 - Getting issue details... STATUS

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

KIP-405 introduced the Tiered Storage feature, which allows to scale the cluster storage independently from CPU and memory and, at the same time, improve maintenance operations. For each leader's partition log, there is a periodic RLMCopyTask that copies inactive and expired segments to the configured remote storage system, along with their auxiliary state (indexes, producer state snapshots, leader-epoch checkpoints). Each remote segment copy is associated with a metadata object, that contains a UUID for that operation. If the same segment copy is retried, a new metadata object with a new UUID is created. By default, these metadata are store in an internal non-compacted topic called __remote_log_metadata, and they include the remote segment state, which is initialized to COPY_SEGMENT_STARTED.

When a segment copy fails with a RemoteStorageException, we immediately try to delete the remote segment data, and the operation will be automatically retried in the next task iteration. If there is a persistent remote storage or network issue, the cleanup operation can also fail, but it is never retried. This leaves the segment in a dangling state, because it never transitions to COPY_SEGMENT_FINISHED. Dangling segments are not counted in the periodic retention size calculation, but their uploaded data accumulate in the remote storage if not manually deleted. This may cause storage exhaustion, or additional expense in case of cloud storage.

Additionally, the current logic is complicated and fragile, because the COPY_SEGMENT_STARTED actually represents more than one segment state.

Dangling state

We propose to introduce a new non-terminal segment state called DANGLING, which would be set if the segment copy fails. At the end of RLMCopyTask.copyLogSegment iteration, the new state will be used to filter all dangling segments and attempt a cleanup deletion transitioning to DELETE_SEGMENT_STARTED state. If the deletion attempt fails, the state will transition back to DANGLING, so that we can retry in the next iteration. After this change, the remote segment state will clearly reflect the actual segment's state, and it won't count against the retention size calculation computed in RLMExpirationTask.cleanupExpiredRemoteLogSegments.

This is the updated state transition diagram:

+--------------------+         +---------------------+
|COPY_SEGMENT_STARTED|-------->|COPY_SEGMENT_FINISHED|
+--+--------------+--+         +--+------------------+
   |              |               |
   |              |               |
   v              v               v
+---+----+     +---+---------------+--+
|DANGLING|---->|DELETE_SEGMENT_STARTED|
+---+----+     +--+--------+----------+
    ^             |        |
    |_____________|        |
                           v
               +-----------+-----------+
               |DELETE_SEGMENT_FINISHED|
               +-----------------------+

Public Interfaces

The RemoteLogSegmentState public enumeration is declared evolving, and it will be changed to add the DANGLING state.

@InterfaceStability.Evolving
public enum RemoteLogSegmentState {

    /**
     * This state indicates that the segment copy failed and the broker will attempt to clean any uploaded data.
     * A segment copy could fail because of a network or remote storage issue.
     */
    DANGLING((byte) 4);

Proposed Changes

We will set the new DANGLING state in RLMCopyTask.copyLogSegment, just before attempting to delete the remote segment that failed the copy operation (catch block).

Right after RLMCopyTask.copyLogSegment we will call a new method called cleanupDanglingRemoteLogSegments to filter all dangling remote segments for that partition using the new DANGLING state, and then attempt a deletion.

New broker metrics will be added to track the current number and total size of all dangling remote segments, while related metrics will be updated.

The documentation will be updated with the new metrics.

Compatibility, Deprecation, and Migration Plan

There is no backwards compatibility issue.

Test Plan

Existing unit tests will be updated as needed, and a new unit test created to verify the dangling remote segment cleanup.

Rejected Alternatives

There is no rejected alternative.