...

So the maximum amount of memory the client can consume is max.partition.fetch.bytes * num_partitions, where num_partitions is the total number of partitions currently being fetched by consumer.

This leads to following problems:

1. Since num_partitions can be quite big (several thousands), the memory required for fetch response responses can be several GB
2. max.partition.fetch.bytes can not be set arbitrarily low since it should be greater than maximum message size for fetch request to work.
3. Memory usage is not easily predictable - it depends on consumer lag

This KIP proposes to introduce new version of fetch request with new parameter response_max_bytes to limit the size of fetch response and solve above problem.

In particular, if consumer issues N parallel fetch requests, the memory consumption will not exceed N * response_max_bytes.

Actually, it will be min(N * response_max_bytes,  max.partition.fetch.bytes * num_partitions)  since per-partition limit is still respected.

Public Interfaces

This KIP introduces:

...