Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...


Essentially, we can use shared storage like S3 or NFS to enhance Kafka's scalability and eliminate cross-AZ traffic. Luckily, the community has made significant efforts to reduce cross-AZ traffic for producers (KIP-1123: Rack-aware partitioning for Kafka Producer) and consumers (KIP-392: Allow consumers to fetch from closest replica, KIP-881: Rack-aware Partition Assignment for Kafka Consumers). With Kafka on shared storage, data replication is unnecessary, eliminating cross-AZ replication traffic. Therefore, this KIP will focus on creating an abstract log layer to facilitate building stateless Kafka on shared storage, enhancing scalability.

...