...
Current state: WIP
Discussion thread:
Voting Discussing thread: here
JIRA:
Motivation
The idea is to keep in the way of saving money getting accurate information about batching and compression.
In our company we have hundreds of clusters and hundreds of marketplaces (producers and consumers) so basically several clients we don't know.
Currently thanks to this KIP-824 now we have a way to inspect the produced requests without affecting the cluster performance. For now we have a rudimentary way which is reading the STDOUT and parsing it in order to get the batching information and the compression type.
...
I would like to create a script which read a sample of the segment log, and output in it into a json if the topic/s will benefit of the batching and compression printing the compression ratio.
...
- Window time in ms to use a as a potential group of batching
...
- Compression algorithm
- Skip not active topics and internal ones
Code Block | ||||
---|---|---|---|---|
| ||||
$ bin/kafka-simpulate-batching-and-compression.sh --topic topic_test --max-bytes 5000000 --batching-window-time 300 --min-records-for-batching 30 --batching-only-by-producer-id --compression-algorithm lz4 --topic-considered-active-last-produce-request-hours 1 --output json |
...
Code Block | ||||
---|---|---|---|---|
| ||||
{ { "topic": "topic_test" { "alrreadyalready_batching": "false", "already_compressing": "false", "candiatecandidate_for_bathcing": "true", "candidate_for_compression": "true", "compression_ratio_percetage": 400, "average_recrods_per_batch": 25 } } } |
...
Then the samples are taken respecting the amount of bytes and from the end of the active segment.
Rejected alternatives
WIP
Compatibility, Deprecation, and Migration Plan
- This is a new script so neither creates any compatibility issue nor migration plan is needed.
- WIP
...