...

• Diskless Topic: A topic which does not append to or serve from block storage devices.
• Object Storage: A shared, durable, concurrent, and eventually consistent storage supporting arbitrary sized byte values and a minimal set of atomic operations: put, delete, list, and ranged get.
• Object Key: A unique reference to an object within Object Storage.
• Batch: A container for Kafka records and a unit of record representation in the network protocol and storage format in both status quo Kafka and diskless topics.
• Shared Log Segment Object: An object containing a shared log segment for one or more diskless topic-partitions on the object storage. Contains record batches similar to classic Kafka topics.
• Batch Coordinate: A reference to a record batch within a shared log segment object, at some byte range.
• Diskless Batch Coordinator: Component serving as source of truth about batch coordinates and log segment objects. Establishes a global order of batches within a diskless topic, and serves coordinates to enable retrieval of batches.
• Object Compaction: Distinct from log compaction. A background asynchronous process which reads from and writes to multiple shared log segment objects to manage already-written objects.

Motivation

KIP-1150: Diskless Topics introduces the concepts of diskless topics, KIP-1163: Diskless Core describes in detail how data is written to and read from diskless topics. According to these KIPs, batches are forever attached to their original objects they were uploaded and committed in. This is problematic because of several reasons:

...