Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Our proposal is in a way also trying to reduce the state size of Kafka broker by uploading more recent log segments onto object storage but we are implementing it in a more conservative way.

Note that we mentioned KIP-1150 in the Rejected Alternative section but we don't view this KIP and KIP-1150 are all-or-nothing alternatives, they are optimized for different use cases (this KIP proposal are optimizing for low-latency acks=1 use case while KIP-1150 is optimizing for a longer latency but diskless implementation), we believe some of the ideas or implementations can be learned or adopted by both KIPs.

Appendix: Cost Estimate for a typical workload

...

See the following table for cost comparison.  The charge is per month per kafka broker


Baseline

S3E1Z

Storage/Instance Cost

$866

$776

Transfer Cost

$1612

$648

Total Cost

$2478

$1424

Latency

Single digit ms

Single digit ms

Across AZ

Yes

Yes


The result is ~43% reduction in cost.

...