This page is meant as a template for writing a KIP. To create a KIP choose Tools->Copy on this page and modify with your content and replace the heading with the next KIP number and a description of your issue. Replace anything in italics with your own description.

Status

Current state: Draft[One of "Under Discussion", "Accepted", "Rejected"]

Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]

JIRA: KAFKA-4133

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

With KIP-74, we now have a good way to limit the size of fetch responses, but it may still be difficult for users to control overall memory since the consumer will send fetches in parallel to all the brokers which own partitions that it is subscribed to. To give users finer control, it might make sense to add a `max.in.flight.fetches` setting to limit the total number of concurrent fetches at any time.

Public Interfaces

The following options will be added to KafkaConfigs.java and can be configured as properties for Kafka clients:

1. max.in.flight.fetches (Integer) Specifies the maximum number of in flight fetches the client can make

Proposed Changes

Update Fetcher.java to take into account the number of existing in-flight fetches before initiating a new fetch request. NetworkClient already tracks the total number of in-flight requests, so we'd just need to check this when creating new fetches in Fetcher.createFetchRequests().

Compatibility, Deprecation, and Migration Plan

• What impact (if any) will there be on existing users?
• If we are changing behavior how will we phase out the older behavior?
• If we need special migration tools, describe them here.
• When will we remove the existing behavior?

Rejected Alternatives

If there are alternative ways of accomplishing the same thing, what were they? The purpose of this section is to motivate why the design is the way it is and not some other way.