...

Object storage implementations are expected to implement caching to reduce the quantity of overall requests to the underlying storage. These caches should be partitioned per-rack, to maintain good locality for requests. Caching is discussed in detail in a follow-up KIP.

Batch deletion

There are various reasons a batch may require deletion:

• the size and/or time retention settings of the topic;
• the partition truncation by the user;
• deletion of a topic.

Since objects are immutable, logical batch deletion doesn’t change the object content. For each object, it’s necessary to track its effectively used size. This size decreases by the batch size when a batch is deleted from the object. When the used size becomes 0, the object could be actually deleted.

Image Added

Object deletion will be implemented as an asynchronous operation. First, the object is marked for deletion by the batch coordinator and a broker performs the actual deletion from the object storage. This has two advantages:

1. The Batch Coordinator doesn’t need to be concerned about the object storage, to have credentials and the code to access it.
2. The deletion could have a grace period. The grace period is a useful way to allow potentially in-progress Fetch requests to successfully finish.

To enforce the time and size retention settings of topics, a background process will periodically check metadata of diskless topics and logically delete affected batches.

It’s quite possible that for compliance reasons, a particular batch has a deadline for physical deletion. Despite the batch being logically deleted, its data are located in an object that may be kept alive by other batches, potentially forever. Therefore, batches with physical deletion deadlines must be either moved to isolated files during merging, or additional merge passes will be necessary to physically delete the data. This problem is addressed in KIP-1165.

Public Interfaces

Plugin Interfaces

...