DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
| Table of Contents |
|---|
Status
Current state: Under Discussion
Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]
JIRA: Jira
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation
...
KIP-405 introduced the Tiered Storage feature, which allows to scale the cluster storage independently from CPU and memory, also improving maintenance operations. For each leader's partition log, there is a periodic RLMCopyTask that copies inactive and locally expired segments to the configured remote storage system, along with their auxiliary state (indexes, producer state snapshots, leader-epoch checkpoints). Each segment copy is associated with a metadata object that contains a UUID for that specific copy operation. By default, these metadata are store in an internal non-compacted topic called __remote_log_metadata, and include the remote segment state, which is initialized to COPY_SEGMENT_STARTED.
...
Public Interfaces
...
The RemoteLogSegmentState public enumeration is declared evolving, and it will be changed to add the DANGLING state.
...
| language | java |
|---|
...
Proposed Changes
...
We propose to introduce a new non-terminal remote segment state called DANGLING, which would be set when the segment copy fail (no deletion will be attempted at this stage). After copying all eligible segments, the RLMCopyTask will try to delete each dangling segment, moving the state to DELETE_SEGMENT_STARTED. If the delete operation fails, the state will move back to DANGLING, so that we can retry in the next RLMCopyTask execution, otherwise the remote segment will move to DELETE_SEGMENT_FINISHED, which is a terminal state.
...
| language | text |
|---|---|
| title | Updated state transition diagram |
...
New broker metrics will be added to track the current number and total size of dangling remote segments.
...
kafka.server:type=BrokerTopicMetrics,name=DanglingRemoteLogCount,topic=([-.\w]+)
...
kafka.server:type=BrokerTopicMetrics,name=DanglingRemoteLogBytes,topic=([-.\w]+)
Compatibility, Deprecation, and Migration Plan
...
Test Plan
...
Rejected Alternatives
...