THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
Public Interfaces
This feature introduces two new options, 'compression.level' and 'the following new options to producer, topic, and broker configuration.
Producer
| Name | Description |
|---|---|
compression.level | Compression level to be used by a producer. |
| compression.buffer.size | The size of compression buffer to be used by a producer. |
Topic and Broker
| Name | Description |
|---|---|
| compression.level | Compression level to be used by a broker, if 'compression.type' is 'none' nor 'producer.' |
| compression.buffer.size | The size of compression buffer to be usedby a broker, |
...
| if 'compression.type' is 'none' nor 'producer.' | |
| compression.[gzip,snappy,lz4, zstd].level | Compression level to be used by a broker, if 'compression.type' is 'producer' and the compression type of produced record is 'gzip', 'snappy', 'lz4', or 'zstd.' |
| compression.[gzip,snappy,lz4, zstd].buffer.size | The size of compression buffer to be usedby a broker, if 'compression.type' is 'producer' and the compression type of produced record is 'gzip', 'snappy', 'lz4', or 'zstd.' |
The valid range and default value of compression level and buffer size are
- If specified option is not supported for the codec ('compression.type'), it is ignored (e.g., compression level for snappy or buffer size for zstd.)
- If specified value is not available (or invalid), it raises an error.
- If there is no specified value, it falls back to the default one.
compression.level
The valid range and the default compression level is entirely up to the compression library, so they may be changed in the future. Their current values are like following:
Compression level
| Compression Codec | availability | Minimum Level | Maximum Level | Valid Range | Current Default |
|---|---|---|---|---|---|
| gzip | Yes | 1 (Deflater.BEST_SPEED) ~ 9 (1)Deflater.BEST_COMPRESSION (9) | 6 | ||
| snappy | No | - | - | - | |
| lz4 | Yes | 1 ~ 17 | 9 | ||
| zstd | Yes | -131072 ~ 22 | 3 |
...
Compression buffer
...
size
...
| Compression Codec | availability | Valid ValuesRange | Default Size | Note |
|---|---|---|---|---|
| gzip | Yes | Positive Integer | 8192 (8kb) | Kafka's own default. |
| snappy | Yes | Positive Integer | 32768 (32kb) | Library default. |
| lz4 | Yes | 4~7 4 ~ 7 (4=64kb, 5=256kb, 6=1mb, 7=4mb) | 4 (64kb) | Kafka's own default. |
| zstd | No | - | - | - |
Proposed Changes
The compression will be done with the new options.
Producer
The record batches will be compressed with the specified level using a compression buffer with specified sizethe specified compression buffer size.
- If the specified option is not supported for the codec ('compression.type'), it is ignored (e.g., compression level for snappy or buffer size for zstd.)
- If the specified value is not available (or invalid), it raises an error.
- If there is no specified value, it falls back to the default one.
Broker
After the offsets are assigned to the record batches sent by the producer, they will be recompressed like following:
- For all options,
- Topic configuration comes first; if it is not set, broker configuration is used. (In other words, topic configuration overrides the broker configuration.)
- If the specified option is not supported for the codec ('compression.type'), it is ignored (e.g., compression level for snappy or buffer size for zstd.)
- If there is no specified value, it falls back to the default one.
- If 'compression.type' is not 'none' or 'producer', the record batches will be recompressed using 'compression.type' codec with 'compression.level' and 'compression.buffer.size.' If either of those values is not available (or invalid), it raises an error.
- If 'compression.type' is 'producer', the record batches will be recompressed using the same codec with corresponding 'compression.[gzip,snappy,lz4,zstd].level' and 'compression.[gzip,snappy,lz4,zstd].buffer.size.' If either of those values is not available (or invalid), it raises an error.
Compatibility, Deprecation, and Migration Plan
...