Page History

...

Server processes partitions in order they appear in request.

If top level max_bytes parameter is Int.MAX_INT ("no limit"), the request is processed exactly as before.

Otherwise, for each partition except the first one server fetches up to corresponding partition limit max_bytes, but not bigger than remaining response limit.

...

This algorithm provides following guarantees:

• FetchRequest with top level max_bytes != Int.MAX_INT always always makes progress - if server has message(s), than at least one message is returned irrespective of max_bytes
• FetchRequest response size will not be bigger than max(max_bytes, size of the first message in first partition)

...

Compatibility, Deprecation, and Migration Plan

New The new fetch request is designed to work properly even if the top level max_bytes is less than the message size. If max_bytes is Int.MAX_INT, new request behaves exactly like old one.

So we can even make this KIP absolutely transparent for users by making setting default for both max.fetch.bytes and replica.fetch.response.max.bytes to Int.MAX_INT.

However, since clients like ReplicaFetcherThread and Java Consumer are ready for new fetch request, we decided to enable   We decided to establish the following defaults:

fetch.max.bytes = 50MB

...

1. Together with addition of global response limit deprecate per-partitions limit. Rejected since per-partition limit can be useful for Kafka streams (see mail list discussion). 
2. Do random partition shuffling on server side. Pros: ensure fairness without client-side modifications. Cons: non-deterministic behaviour on server side; round-robin can be easily implemented on client side.