DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
| Table of Contents |
|---|
Status
Current state: [VotingDiscussion]
Discussion thread: here []
...
- Compatibility between an existing codec and a corresponding accelerated codec.
- The default behavior with this feature disabled is unchanged
- Correct error handling in failure scenarios
- Multiple compression service types can be correctly used and managed simultaneously
- Validate compression options (e.g compression levels) are honored were applicable
Performance Impact
This feature was tested using a custom deflate compressor plugin that leverages the QAT compression accelerator. The performance was tested against the built-in gzip and zstd Kafka compressors as well as zlib-ng. This QAT-deflate plugin was developed using the QAT-java library. zlib-ng data was collected by using LD_PRELOAD and zlib-ng version 1.3.1. Benchmarking shows QAT-deflate results in
- almost 6x gain in throughput relative to gzip
- up to 4x gain in throughput relative to zlib-ng
- about 40% gain in throughput relative to zstd
| Compression Type | Normalized Throughput at SLA (20ms) | compression ratio |
|---|---|---|
gzip | 1 | 0.446 |
| QAT-deflate | 5.93 | 0.442 |
| zlib-ng | 1.41 | 0.441 |
| zstd | 3.97 | 0.442 |
Environment
- Kafka 4.2.0 snapshot
- OpenJDK 21
- Ubuntu 24.04.3 LTS
- AWS ec2 i7i-metal-24xl broker (2 Kafka brokers)
- AWS ec2 m7i-metal-48xl producer
- QATjava-2.3.2, Qatzip-1.3.1
- Producer performance test
- Batch size = 112K
- Record size = 1024 bytes
- replication = 2
- Security protocol = SSL
Notes
- Each producer runs at a fixed throughput of 50K record per second. The number of producers is scaled up until the p99 latency reaches 20ms and the combined throughput is the KPI.
- Consumers are not included in this benchmarking.
- compression ratios are as reported by the producer performance test.
- These tests were run with brokers on a single node and producers on a separate node.
- QAT is only enabled on metal instances (on AWS) which are relatively expensive. Data for gzip, zlib-ng and Zstd were also collected on smaller instances to match costs (6 X i7i-4xl instances). Performance on the smaller instances, while different, does not change the relative performance results.