...
Two microbenchmarks were run to see how each configuration option interacts with the other one; each benchmark measured the read or write ops/sec for various compression options.
Gzip
(Left: Gzip (Write ops/sec), Right: Gzip (Read ops/sec))
Compression level and buffer size did not significantly impact both writing and reading speed; The most significant factor was data size. The impact of compression parameters was restricted only when the data size was very small, and made only ~11% of differences.
Snappy
(Left: Snappy (Write ops/sec), Right: Snappy (Read ops/sec))
Changing the block size did not drastically affect the writing speed - however, if you stick to the too-small block size for large records, it will degrade the writing speed. For example, using a block size of 1024 with 100 bytes-sized data degrades the compression speed 0.01% only. In contrast, if the data size is 10000 or 100000 bytes, the gap in writing speed is 30% or 38%, respectively.
If the record size is so small (like less than 1kb), reducing the block size to smaller than 32kb (default) would be reasonable for both of compression speed and memory footprint. However, if the record size is not so small yet, reducing the block size is not recommended.
Block size also had some impact on reading speed; As the size of data increases, small block size degraded the read performance.
LZ4
(Left: LZ4 (Write ops/sec), Right: LZ4 (Read ops/sec))
Level 9 (default) always did the best; The differences between the compression level were neglectable, and as the block size increased, and in these cases, level 9 also showed similar speed with the other levels.
Overall, as the block size increased, the writing speed decreased for all data sizes. In contrast, the reading speed was almost the same at <= 1.000 bytes. If the block size is too big, the ops/sec for reading operation also degrades.
Zstd
(Left: Zstd (Write ops/sec), Right: Zstd (Read ops/sec))
For the writing speed, as the data size increases, the ops/sec differences between the levels disappear. Disabling the long mode was better than enabling, but if the long mode is turned on, the windowLog of 16 resulted in the best.
...
Real World Dataset
A benchmark with a real-world dataset was run to see the actual compression ratio and speed.
...
The dataset used consists of json data files collected from IoT sensors and measured the average compressed size / elapsed time of 100 batches, each consisting of 10 json files from the dataset.
(Compressed Size in bytes, overall)
(Elapsed in sec, overall)
Gzip
(Gzip Elapsed time, detail)
Gzip took second place in terms of the compression ratio. However, although the variance in the compressed size is less than 15%, as we took the higher compression level, the elapsed time also increases, regardless of the buffer size. So, setting the level to 1 and assigning enough buffer to the consumer would be a preferable strategy in terms of producing and consuming both.
Snappy
(Snappy Elapsed time, detail)
Snappy took the bottom in terms of the compression ratio. The bigger block size snappy uses, both of the compression ratio and speed increased. So, finding the optimal block size seems to be the essential thing in tuning snappy.
LZ4
(LZ4 Elapsed time, detail)
The third place in compression ratio. The default configuration (level = 9, block = 4) outperformed all the other configurations in speed and size both.
...
In general, when the user has to use LZ4, using the default configuration would be the most reasonable strategy. But, if the user needs the smaller compressed data size, enlarging the block size and lowering the level would be an option.
Zstd
(Zstd Elapsed time, detail)
Zstd outperformed all the other configurations, both in compression ratio and speed. Comparing gzip / level=9 / buffer=32768 and zstd / level=1 / window=0, the best configuration among all the other and the most speed-first strategy in zstd, zstd outplayed gzip with only 90% of the size and 7% of compression time.
...