Versions Compared

Old Version 3

changes.mady.by.user Ivan Yurchenko

Saved on

compared with

New Version Current

changes.mady.by.user Greg Harris

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Discard KIP due to 1163 design change.

👷 WORK IN PROGRESS

Authors: Greg Harris, Ivan Yurchenko, Jorge Quilcate, Giuseppe Lillo, Anatolii Popov, Juha Mynttinen, Josep Prat, Filip Yonov

Table of Contents

Status

Current state: Under DiscussionDiscarded

Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]

JIRA: here [Change the link from KAFKA-1 to your own ticket]19161

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

This KIP was discarded due to a design change in KIP-1163 which made it unnecessary.

Glossary

  • Diskless Topic: A topic which does not append to or serve from block storage devices.
  • Object Storage: A shared, durable, concurrent, and eventually consistent storage supporting arbitrary sized byte values and a minimal set of atomic operations: put, delete, list, and ranged get.
  • Object Key: A unique reference to an object within Object Storage.
  • Batch: A container for Kafka records and a unit of record representation in the network protocol and storage format in both status quo Kafka and diskless topics.
  • Shared Log Segment Object: An object containing a shared log segment for one or more diskless topic-partitions on the object storage. Contains record batches similar to classic Kafka topics.
  • Batch Coordinate: A reference to a record batch within a shared log segment object, at some byte range.
  • Diskless Batch Coordinator: Component serving as source of truth about batch coordinates and log segment objects. Establishes a global order of batches within a diskless topic, and serves coordinates to enable retrieval of batches.
  • Object Compaction: Distinct from log compaction. A background asynchronous process which reads from and writes to multiple shared log segment objects to manage already-written objects.

Motivation

KIP-1150: Diskless Topics introduces the concepts of diskless topics, KIP-1163: Diskless Core describes in detail how data is written to and read from diskless topics. According to these KIPs, batches are forever attached to their original objects they were uploaded and committed in. This is problematic because of several reasons:

...