...

org.apache.kafka.common.compress.ZstdCompression.java

org.apache.kafka.common.utils.ByteBufferOutputStream

org.apache.kafka.common.utils.BufferSupplier

org.apache.kafka.common.utils.ByteBufferInputStream

org.apache.kafka.common.utils.ChunkedByteStream

org.apache.kafka.common.record.CompressionType

>> sample Producer.properties file >>

compression.type=gzip
compression.gzip.level=1
compression.gzip.provider=com.intel.qat.kafka.QatDeflateCompression$Builder

Proposed Changes

This KIP leverages Java’s Service Loader API to offload compression/decompression for a specific codec to a provided alternative. Each existing compression codec (specifically its builder) represents a service. A service provider is any alternative implementation of the specific service (codec). On startup the service loader will attempt to load any discovered service providers for each service and will potentially replace the base implementation with the provided implementation. As an example, GzipCompression$Builder is one of four services. A service provider would have to ‘register’ itself as a provider of this service and would return a GzipCompression$Builder when loaded which would be used instead of the base implementation. An optional configuration ‘compression.[gzip].service’ exists in the Producer and Server configurations that indicates a preferred service provider to be loaded if found. Any new service provider should extend the base implementation and thus return the same “CompressionType”. A new method isAvailable() will be added to the “Compression$Builder” interface that returns true if the accelerated codec is available at runtime.
The modifications to the base implementations include static initializers to load service providers and a static method to return a builder from the base implementation of service provider.
 

...