...

• Write through to object storage, avoiding local disk usageusing local disks for durable storage
• Pick pluggable commodity storage backends based on their environment
• Balance traffic among brokers and eliminate broker hotspots with per-client granularity
• Upgrade and scale clusters without moving active segments or electing leaders
• Tradeoff cost optimization and latency on a per-topic basis

...

It's important to clarify what exactly "diskless" means. "Diskless" primarily refers to not using broker disk for storing disks as the primary durable storage of user data. There are no index files stored on broker disk for diskless topics as well. However, diskless topics still require some broker disk usage, particularly:

• normal topic KRaft metadata;
• batch metadata may be stored on broker disk depending on the batch coordinator implementation (e.g. in a Kafka topic);
• brokers may require some limited amount of disk space to perform certain operations like object compaction;
• diskless topic user data may be stored on disk while being copied to tiered storage
• diskless topic user data may be cached on disk to be served to consumerscaching in the read path may optionally use broker disk instead of memory.
  

It is also worth mentioning that Diskless topics are not meant to change the Kafka Storage API, but to have a separate request processing that takes care of the access to remote storage.

In short, Diskless is to “No Disks” as Serverless is to “No Servers,” the attached disks become a less important abstraction for operators but are still functionally present.

Proposed Changes

This KIP will not require any changes to the codebase or documentation upon acceptance. By accepting this KIP, we will come to a consensus on the need for this feature, and its end-user requirements, but not any specific implementation details.

For details on the planned implementation, please see the integral follow-up KIPs:

• KIP-1163: Diskless Core
• KIP-1165: Object Compaction for DisklessKIP-1164: Topic Based Batch Coordinator
• KIP-1181: Metadata Rack Awareness for Diskless TopicsKIP-F: Cache Strategy
• KIP-O: Garbage collection for Diskless objects

...

• Broker Roles: Specializing brokers between produce/consume/coordination/compaction operations and permitting heterogeneous Kafka clusters
• Parallel Produce Handling: Processing multiple Produce requests concurrently, increasing potential throughput in high latency environments.Transactions on Diskless Topics: Including Diskless Topics in Exactly-Once Semantics Workloads
• Iceberg Format: Allowing massively parallel processing of at-rest topic data. This work enables a pluggable storage interface where one can innovate in the log format layer independently
• Dynamically Enabled Diskless: Allowing extremely easy migrations to try out & revert DisklessUnification/Relationship with Tiered Storage: Identifying a long-term vision for Diskless and Tiered Storage plugins
• Multi-region active-active topics with automatic failover

...