...

• the local size and/or time retention settings of the topic;
• the partition truncation by the user;
• deletion of a topic.

Since objects are immutable, the coordinator metadata will allow logical batch deletion doesn’t change , without changing the object content. To enforce the time and size retention settings of topics, a background process within the diskless coordinator will periodically check metadata of diskless topics and logically delete affected batches. When topic retention settings cause segments to be written to Tiered Storage, the batches contained in those segments are also logically deleted from the diskless coordinator.

For each object, it’s necessary to track its effectively used size. This size decreases by the batch size when a batch is deleted from the object. When the used size becomes 0, the object could be actually deleted.

...

1. The Diskless Coordinator doesn’t need to be concerned about the object storage, to have credentials and the code to access it.
2. The deletion could have a grace period. The grace period is a useful way to allow potentially in-progress Fetch requests to successfully finish.

To enforce the time and size retention settings of topics, a background process will periodically check metadata of diskless topics and logically delete affected batches.

1. A single object may contain live batches for other coordinators, and the node performing the deletion must contact other coordinators in order to prove the object is safe to delete.
2. Nodes can periodically reconcile the list of objects in the storage to the list of WAL Segments in the diskless coordinators, and delete orphaned objects that were not properly committed at some earlier time.

 It’s quite possible that for compliance reasons, a particular batch has a deadline for physical deletion. Despite the batch being logically deleted, its data are is located in an object that may be kept alive by other batches, potentially forever. Therefore , batches with physical deletion deadlines must be either moved to isolated files during merging, or additional merge passes will be necessary to physically delete the data. This problem is addressed in KIP-1165it is necessary to regularly move batches from WAL Segments to Tiered Storage segments. In practical terms, it means that the lifetime of a WAL segment is slightly more than the longest configured roll time, and the earliest physical deletion guarantee is after the longest roll.

For example: Topic A rolls a segment and uploads to tiered storage once per hour, and Topic B has a 15 minute retention time. WAL Segments may contain data from both Topic A and B together. 15 minutes after a batch is produced to Topic B, it is logically deleted and no longer visible to consumers. 1 hour after a batch is produced to Topic A, it is rolled and copied to Tiered Storage. Then both batches from topic A and B can be physically deleted and the space reclaimed.

Public Interfaces

Plugin Interfaces: Storage Backend

...