...

Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]

JIRA:here 

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

...

Output

The tool will print results to standard out. The tool reports information about the batches in the log segment (as more batching often helps improve the effectiveness of compression), the breakdown of compression types found in the log segment, and the results of applying each compression type. A sample output:

Original log size: 536793767 bytes
Uncompressed log size: 536793767 bytes

Batch stats:
  16593/20220 batches contain >1 message
  Avg number of messages per batch: 3.68
  Avg batch size (original): 5180 bytes
  Avg batch size (uncompressed): 5180 bytes

Number of input batches by compression type:
  none: 20220

COMPRESSION-TYPE  COMPRESSED-SIZE  COMPRESSION-RATIO  SPACE-SAVINGS  AVG-RATIO/BATCH  TOTAL-TIME        SPEED
gzip                    118159324              4.543         22.01%            1.795     13875ms   36.90 MB/s
snappy                  160597012              3.342         29.92%            1.549      2678ms  191.16 MB/s 
zstdlz4                     112737048161711232              43.761319         2130.00%13%            1.775576      5103ms2616ms  100195.3269 MB/s
lz4zstd                    112737048 161711232              34.319761         3021.13%00%            1.576775      2616ms5103ms  195100.6932 MB/s

Breakdown of outputs:

Compression Type - the configured compression type
Compressed Size - size in bytes of the log segment after compression
Compression Ratio - the ratio of the uncompressed size (or original size, if --no-recompression is set) to the compressed size
Space Savings - the reduction in size relative to the uncompressed size (or original size, if --no-recompression is set)
Avg Ratio/Batch - the mean compression ratio on a per-batch basis
Time - how long it took to compress all batches for the given compression type
Speed - the average rate at which the compression type is able to compress the log segment

...

kafka-compression-analyzer.sh aims to compress messages in the same manner a producer would and record the different in size of each batch. The tool sequentially iterates over each RecordBatch in a log file (very similar to kafka-dump-log.sh), compresses it into a new MemoryRecords object for each compression type supported by Kafka, and records the sizes size of the batch both before and after compression. Since the tool only compresses existing batches as they were written to the log file and does not merge or split them, the tool effectively measures the resulting log size as if compression were enabled across all producers, without any other producer configurations being changed (ex. linger.ms).

If a RecordBatch is already compressed in the log, by default the tool will decompress the batch and then recompress it using the other compression types. This allows the tool to report the resulting size of the log as if all RecordBatches are compressed using each compression type. This can be disabled via the --no-recompression flag, in which case compression will only be done on uncompressed batches. Therefore, results with the --no-recompression flag will effectively show the impact of compression if all producers currently using compression.type=none were configured to use a given were to be normalized to use a single compression type.

Notes

• The shell script will run kafka.tools.LogCompressionAnalyzer, which contains the source of the tool
• There is precedent for read-only tools that operate on log files (i.e. kafka-dump-log.sh), if there are any consequences of running this tool on a log file the concerns on a broker would be shared by those tools
• The tool does not spawn multiple threads
• The tool will likely consume an entire core while running
  • Consider copying the log segment and running the tool on a non-broker machine to avoid starving the broker of CPU

...

This proposal adds a new tool and changes no existing functionality.

Potential Future Work

There may be situations where it is not desirable for all batches to be compressed with a single compression type. For this reason, it may eventually be useful to provide a way to restrict the batches being compressed for the analysis. For example, it might be possible to exclude batches compressed with a certain compression type from being recompressed, only analyzing the remaining subset of the log. However, this can be implemented as a follow-up addition once better motivation for what mechanisms are needed and how they might work is available.

Rejected Alternatives

TODO