...

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Currently the only possible way for client to limit fetch response size is via per-partition response limit taken from config setting max.partition.fetch.bytes.

...

This KIP proposes to introduce new version of fetch request with new parameter "limit_bytes" to limit the size of fetch response and solve above problem. The per-partition limit is removed from fetch request.

Public Interfaces

This KIP introduces:

• New fetch request (v.3) with a global response limit and without per-partition limit
• New client-side config parameter fetch.limit.bytes - global fetch size limit
• New replication config parameter replica.fetch.limit.bytes - limit used by replication thread
• New server-side config parameter fetch.partition.max.bytes - maximum per-partition server side limit when serving new fetch request
• New inter-broker protocol version "0.11.0-IV0" - starting from this version brokers will use fetch request v.3 for replication 

Proposed Changes

Proposed changes are quite straightforward. New fetch request processes partitions in order they appear in request.

...

This way we can ensure that response size is less than (limit_bytes + message.max.bytes).

Caveats

Since new fetch request processes partitions in order and stops fetching data when response limit is hit, client should use some kind of partition shuffling to ensure fairness.

...

Old fetch requests should be processed on server exactly as before this KIP.

Discussed/Rejected Alternatives

Some discussed/rejected alternatives:

...