Page History

...

This KIP proposes to introduce new version of fetch request with new top-level parameter response_max_bytes to limit the size of fetch response and solve above problem.

In particular, if consumer issues N parallel fetch requests, the memory consumption will not exceed N * response_ max_bytes.

Actually, it will be min(N * response_ max_bytes,  max.partition.fetch.bytes * num_partitions)  since per-partition limit is still respected.

...

Proposed changes are quite straightforward. We introduce FetchRequest v.3 with new top level parameter response_max_bytes:

Fetch Request (Version: 3) => replica_id max_wait_time min_bytes response_max_bytes [topics]
  replica_id => INT32 
  max_wait_time => INT32
  min_bytes => INT32
  response_max_bytes => INT32
  topics => topic [partitions]
    topic => STRING
    partitions => partition fetch_offset max_bytes
      partition => INT32
      fetch_offset => INT64
      max_bytes => INT32

...

Server processes partitions in order they appear in request.

If response_top level max_bytes parameter is Int.MAX_INT ("no limit"), the request is processed exactly as before.

...

This algorithm provides following guarantees:

• FetchRequest with response_top level max_bytes != Int.MAX_INT always makes progress - if server has message(s), than at least one message is returned irrespective of response_max_bytes
• FetchRequest response size will not be bigger than max(response_max_bytes, size of the first message in first partition)

...

New fetch request is designed to work properly even if response_top level max_bytes is less than message size. If response_max_bytes is Int.MAX_INT, new request behaves exactly like old one.

...