Page History

...

This proposal suggests adding compression level and compression buffer size options to the producer and broker config.

Basically, CPU (running time) and I/O (compressed size) are trade-offs in compression. Since the best is use case dependent, lots of compression algorithms provide a way to control the compression level with reasonable default level, which results in a good performance in general. Add to this, the compression ratio is also affected by the buffer size - some users may want to trade compression for less/more memory.

...

This feature introduces two new options, 'compression.level' and 'compression.buffer.size,' , to the producer and broker configuration.

• If specified option is not supported for the codec ('compression.type'), it is ignored (e.g., compression level for snappy or buffer size for zstd.)
• If specified value is not available (or invalid), it raises an error.
• If there is no specified value, it falls back to the default one.

compression.level

The 'default' column in the following table means 'which value instructs the codec to use the default level,' not a specific level. The valid range and the default compression level is entirely up to the compression library; in other words, specifying -1 with gzip has exactly the same impacts with specifying any value to the 'compression.level'., so they may be changed in the future.

compression.buffer.size

The default compression buffer size follows current implementations; currently, snappy uses the library default, but gzip and lz4 uses Kafka's own default.

...