Current state: [Under Discussion]
Discussion thread: here []
JIRA: here []
Compression improves network and disk utilization and leads to better cluster throughput at the expense of latency. Several compression algorithms (and compression levels) exist which trade off compression ratio for speed. With the use of accelerators, we can speed up compression (improve latency) without affecting compression ratio too much.
Kafka currently supports 4 compression algorithms; Gzip, Zstd, Snappy and LZ4. This KIP proposes a framework that supports the acceleration of any of these algorithms in Kafka with any hardware accelerator provider. This KIP does not introduce a new compression algorithm but instead allows for compression acceleration on producers, brokers or consumers. Compression, decompression and recompression can be independently accelerated. This means that the accelerated compression strategy MUST be compatible with existing strategies.
This KIP also introduces an accelerated GzipCompressor that uses the QAT (https://www.intel.com/content/www/us/en/products/docs/accelerator-engines/what-is-intel-qat.html) hardware accelerator
org.apache.kafka.common.compress.Compression.java
A new compression class for every compression algorithm to be accelerated by a specific accelerator. Discovery of accelerated codecs will be determined by the presence of the new “Compression” class. This new compression class should extend the existing codec to be accelerated and thus return the same “CompressionType”. A new method isAvailable() will be added to the interface to determine if the accelerated codec is available at runtime. The appropriate codec will be ‘selected’ at runtime based on a preset order of priority if more than one option is available.
A specific accelerated codec must be compatible with the codec it is accelerating in all directions. Since compression and decompression typically occur in separate processes, the accelerated codec need not be available in all nodes of the cluster for compression/decompression to occur correctly. This KIP does not affect the message format and is purely opportunistic; Compression and decompression will continue to operate normally even if no accelerators are present
Tests must prove compatibility between an existing codec and a corresponding accelerated codec.
See (KIP-984: Add pluggable compression interface to Kafka)