...

Current state: [Under Discussion]

Discussion thread: here []

JIRA: here []

Motivation

Compression improves network and disk utilization and leads to better cluster throughput at the expense of latency. Several compression algorithms (and compression levels) exist which trade off compression ratio for speed. With the use of accelerators, we can speed up compression (improve latency) without affecting compression ratio too much. 
Kafka currently supports 4 compression algorithms; Gzip, Zstd, Snappy and LZ4. This KIP proposes a framework that supports the acceleration of any of these algorithms in Kafka with any hardware accelerator provider. This KIP does not introduce a new compression algorithm but instead allows for compression acceleration on producers, brokers or consumers. Compression, decompression and recompression can be independently accelerated. This means that the accelerated compression strategy MUST be compatible with existing strategies.
This KIP also introduces an accelerated GzipCompressor that uses the QAT (https://www.intel.com/content/www/us/en/products/docs/accelerator-engines/what-is-intel-qat.html) hardware accelerator

...