...
This KIP proposes to introduce new version of fetch request with new top-level parameter response_max_bytes to limit the size of fetch response and solve above problem.
In particular, if consumer issues N parallel fetch requests, the memory consumption will not exceed N * response_ max_bytes.
Actually, it will be min(N * response_ max_bytes, max.partition.fetch.bytes * num_partitions) since per-partition limit is still respected.
...
Proposed changes are quite straightforward. We introduce FetchRequest v.3 with new top level parameter response_max_bytes:
Fetch Request (Version: 3) => replica_id max_wait_time min_bytes response_max_bytes [topics]
|
...
Server processes partitions in order they appear in request.
If response_top level max_bytes parameter is Int.MAX_INT ("no limit"), the request is processed exactly as before.
...
This algorithm provides following guarantees:
- FetchRequest with response_top level max_bytes != Int.MAX_INT always makes progress - if server has message(s), than at least one message is returned irrespective of response_max_bytes
- FetchRequest response size will not be bigger than max(response_max_bytes, size of the first message in first partition)
...
New fetch request is designed to work properly even if response_top level max_bytes is less than message size. If response_max_bytes is Int.MAX_INT, new request behaves exactly like old one.
...