This page is meant as a template for writing a KIP. To create a KIP choose Tools->Copy on this page and modify with your content and replace the heading with the next KIP number and a description of your issue. Replace anything in italics with your own description.
Status
Current state: Draft[One of "Under Discussion", "Accepted", "Rejected"]
Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]
JIRA: KAFKA-4133
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation
With KIP-74, we now have a good way to limit the size of fetch responses, but it may still be difficult for users to control overall memory since the consumer will send fetches in parallel to all the brokers which own partitions that it is subscribed to. To give users finer control, it might make sense to add a `max.in.flight.fetches
` setting to limit the total number of concurrent fetches at any time.
Public Interfaces
The following options will be added to KafkaConfigs.java
and can be configured as properties for Kafka clients:
max.in.flight.fetches
(Integer) Specifies the maximum number of in flight fetches the client can make
Proposed Changes
Update Fetcher.java to take into account the number of existing in-flight fetches before initiating a new fetch request. NetworkClient already tracks the total number of in-flight requests, so we'd just need to check this when creating new fetches in Fetcher.createFetchRequests().
Compatibility, Deprecation, and Migration Plan
This KIP should be transparent to users not interested in setting this new configuration.
Rejected Alternatives
None so far