...

In our company we have hundreds of clusters and hundreds of marketplaces (producers and consumers) so basically several clients we don't know.

Currently thanks to this KIP now we have a way to inspect the produced requests without affecting the cluster performance. For now we have a rudimentary way which is reading the STDOUT and parsing it in order to get the batching information and the compression type.

...

I would like to create a script which read a sample of the segment log, and output in it into a json if the topic/s will benefit of the batching and compression printing the compression ratio.

...

- Window time in ms to use as a a potential group of batching

...

- Compression algorithm  

- Skip not active topics and internal ones

$ bin/kafka-simpulate-batching-and-compression.sh --topic topic_test --max-bytes 5000000 --batching-window-time 300 --min-records-for-batching 30 --batching-only-by-producer-id --compression-algorithm lz4  --topic-considered-active-last-produce-request-hours 1 --output json

...

Then the samples are taken respecting the amount of bytes and from the end of the active segment.

Rejected alternatives

WIP

Compatibility, Deprecation, and Migration Plan

• This is a new script so neither creates any compatibility issue nor migration plan is needed.
• WIP

...