...

Public Interfaces

This feature introduces the following new options for compression level and buffer size to the producer, topic, and broker configuration. About public interfaces, there are two available alternatives:

Type A: Traditional Style

This approach introduces two config entities: compression.level and compression.buffer.size.

...

...

compression.type=gzip
compression.level

...

=4				# NEW: Compression level to be used.
compression.buffer.size

...

=8192	# NEW: The size of compression buffer to be used.

For available values per compression type, please refer 'Compression level' and 'Compression buffer size' subsection under 'Proposed Changes'.

Pros:

1. Easy to adapt.

Cons:

1. It adds two configuration entities and even may require more entities following compression codecs' update.
2. Every time the user tries to change the compression type, they must update `compression.level` and `compression.buffer.size` accordingly. In other words, it is error-prone.

Type B: Key-Value Style (proposed by Becket Qin)

This approach introduces only one config entity, compression.config. It contains a list of KEY:VALUEs concatenated with comma, like listener.security.protocol.map or max.connections.per.ip.overrides.

compression.type=gzip
compression.config=gzip.level:4,gzip.buffer.size:8192,snappy.buffer.size:32768,lz4.level:9,lz4.buffer.size:4,zstd.level:3

The available KEYs are: gzip.level, gzip.buffer.size, snappy.buffer.size, lz4.level, lz4.buffer.size, zstd.level. For available values per each key, please refer 'Compression level' and 'Compression buffer size' subsection under 'Proposed Changes'.

Pros:

1. Adds only one configuration entity, and easy to following compression codecs' updates - If there is a new configuration option for compression codec,
2. Easy to switch compression type: Kafka broker picks up appropriate configuration following the compression type.

Cons:

1. A little bit complicated for initial set up.

Proposed Changes

The compression will be done with the specified level using the specified compression buffer size, like the following:

• If the specified option is not supported for the codec ('compression.type'), it is ignored (e.g., compression level for snappy or buffer size for zstd.)
• If the specified value is not available (or invalid), it raises an error.
• If there is no specified value, it falls back to the default one.

The

...

The valid range and default value of compression level and buffer size are entirely up to the compression library, so they may be changed in the future. Their As of June 2019, their current values are like the following:

Compression level

...

Proposed Changes

The compression will be done with the specified level using the specified compression buffer size, like following:

...

Compatibility, Deprecation, and Migration Plan

...