Versions Compared

Old Version 11

changes.mady.by.user Unknown User (jeqo)

Saved on

compared with

New Version 12

changes.mady.by.user Ivan Yurchenko

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Remove mention of non-support of transactions

...

  1. Data in diskless topics is durably stored solely in object storage, and not in segments on broker disks. 
    1. Classic concepts of log directories, segment rolling, object writing and deletion, message flushing, tiered storage, etc. and the related APIs do not apply. Effectively, brokers will not create any files related to diskless topics.
    2. Remote storage may have higher latency than local disks, increasing the latency of Kafka requests and end-to-end data latency.
  2. Kafka delegates replication of diskless topics to object storage, and does not perform replication itself.
    1. Classic concepts of replicas, replication, ISR, partition assignment and the related APIs do not apply.
    2. All operators can use efficient types of storage backends, such as ones with erasure coding.
    3. Hyperscaler operators can avoid most inter-zone data replication costs.
  3. All brokers are capable of interacting with all diskless topics, and no broker is uniquely considered the leader of a partition.
    1. Classic replica leader election is not applicable to diskless topics and all requests may use any broker as if it were the leader.
    2. Clusters are able to perform fine-grained client balancing across the cluster independently of topic/partition hot spots.
    3. Hyperscaler operators can avoid most inter-zone data ingress/egress costs.

This KIP allows certain a non-fundamental functional limitations limitation between diskless and classic topics: Diskless topics will not immediately support compaction

...

.

The preliminary research suggests that these limitations are not fundamental and can be resolved. They are out of scope of this KIP, and are expected to be addressed in follow-up KIPs. See the Further Work section for more details.

In other functional aspects, diskless topics are indistinguishable from classic topics. This includes durability guarantees, ordering guarantees, transactional and non-transactional producer API, consumer API, consumer groups, share groups, data retention, and authorization mechanisms. Streaming applications can be moved from classic to diskless topics without changing client versions, application code, or experiencing correctness problems.

...