...

Currently, Apache Kafka is often operated in cloud hyperscaler environments where high-reliability object storage is available and more cost-effective than block storage for equivalent workloads. The existing Tiered Storage feature (KIP-405) provides the capability to use object storage for inactive segments, and has seen widespread adoption. However, Tiered Storage does not remove the need for replication of active segments, which is the most substantial infrastructure cost for Apache Kafka operators on hyperscalers today.

Two out of the 3 major clouds charge for cross-availability-zone traffic:

• AWS charges 0.02$ per GiB https://aws.amazon.com/ec2/pricing/on-demand/#Data_Transfer_within_the_same_AWS_Region 
• Google Cloud charges 0.01$ per GiB https://cloud.google.com/vpc/network-pricing?hl=en 
• Azure does not charge cross-AZ traffic

Even in the case where network traffic is not accounted; we believe that the operational benefits of Diskless topics are still appealing to Kafka users.

Multiple protocol-compatible alternatives to Apache Kafka now use object storage to fully replace direct replication and substantially lower the cost to operate a cluster on a hyperscaler cloud. These alternatives are finding market success and their adoption is rising, showing a general market interest in this optimization.

...

• normal topic KRaft metadata;
• batch metadata may be stored on broker disk depending on the batch coordinator implementation (e.g. in a Kafka topic);
• brokers may require some limited amount of disk space to perform certain operations like object compaction;
• caching in the read path may optionally use broker disk instead of memory.
  

It is also worth to mention that Diskless topics are not meant to change the Kafka Storage API, but to have a separate request processing that takes care of the access to remote storage.

Proposed Changes

This KIP will not require any changes to the codebase or documentation upon acceptance. By accepting this KIP, we will come to a consensus on the need for this feature, and its end-user requirements, but not any specific implementation details.

...