Page History

...

But running Kafka on shared storage services is challenging because its classic ISR-based storage engine is designed for local disks. For example, although Kafka can be deployed on file storage services like NetApp Files, it can't benefit from the durability and elasticity of shared file storage.

At AutoMQ, our goal is This KIP aims to make Kafka stateless and enable , allowing it to run seamlessly on shared storage systems , and transforming its architecture from shared nothing to shared storage. But this KIP does not plan to merge AutoMQ into Kafka directly, as it's a major decision for the Kafka community and would require significant resources.This KIP aims to introduce It introduces a unified log layer for deploying Kafka on various shared storage mediums, with offering the following benefits:

• A unified log layer that supports both current replication-based local file storage and shared storage simultaneously.

• Greater flexibility for major users or vendors to deploy Kafka on their own shared storage, as many large companies have their own DFS or HDFS infrastructure.

• Prevent community fragmentation, allowing users or vendors to extend the storage layer without complex modifications to Kafka, fostering greater community collaboration.

...

Essentially, we can use shared storage like S3 or NFS to enhance Kafka's scalability and eliminate cross-AZ traffic. Luckily, the community has made significant efforts to reduce cross-AZ traffic for producers (KIP-1123: Rack-aware partitioning for Kafka Producer) and consumers (KIP-392: Allow consumers to fetch from closest replica, KIP-881: Rack-aware Partition Assignment for Kafka Consumers). With Kafka on shared storage, data replication is unnecessary, eliminating cross-AZ replication traffic. Therefore, this KIP will focus on creating an abstract log layer to facilitate building stateless Kafka on shared storage, enhancing scalability.

...

Currently, Kafka's storage engine heavily relies on UnifiedLog, which includes LocalLog and RemoteLog, and some logic even depends on local files. This makes current storage unsuitable for shared storage support.Image Removed

Image Added

This KIP proposes a new inheritance system for log classes:

...