THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
This feature introduces the following new options to producer, topic, and broker configuration.
Producer
...
compression.level
...
Topic and Broker
| Name | Description |
|---|---|
| compression.level | Compression level to be used by a broker, if 'compression.type' is 'none' nor 'producer.' |
| compression.buffer.size | The size of compression buffer to be usedby a broker, if 'compression.type' is 'none' nor 'producer.'|
compression .[gzip,lz4, zstd].level | Compression level to be used by a broker, if 'compression.type' is 'producer' and the compression type of produced record is 'gzip', 'lz4', or 'zstd.' |
| compression.[gzip,snappy,lz4].buffer.size | The size of compression buffer to be usedby a broker, if 'compression.type' is 'producer' and the compression type of produced record is 'gzip', 'snappy', or 'lz4.' |
The valid range and default value of compression level and buffer size are entirely up to the compression library, so they may be changed in the future. Their current values are like following:
...
The compression will be done with the new options.
Producer
The record batches will be compressed with the specified the specified level using the specified compression buffer size., like following:
- If the specified option is not supported for the codec ('compression.type'), it is ignored (e.g., compression level for snappy or buffer size for zstd.)
- If the specified value is not available (or invalid), it raises an error.
- If there is no specified value, it falls back to the default one.
Broker
After the offsets are assigned to the record batches sent by the producer, they will be recompressed like following:
- For all options,
- Topic configuration comes first; if it is not set, broker configuration is used. (In other words, topic configuration overrides the broker configuration.) If the specified option is not supported for the codec ('compression.type'), it is ignored (e.g., compression level for snappy or buffer size for zstd.)
- If there is no specified value, it falls back to the default one.
- If 'compression.type' is not 'none' or 'producer', the record batches will be recompressed using 'compression.type' codec with 'compression.level' and 'compression.buffer.size.' If either of those values is not available (or invalid), it raises an error.
- If 'compression.type' is 'producer', the record batches will be recompressed using the same codec with corresponding 'compression.[gzip,snappy,lz4,zstd].level' and 'compression.[gzip,snappy,lz4,zstd].buffer.size.' If either of those values is not available (or invalid), it raises an error.
Compatibility, Deprecation, and Migration Plan
...