Status
Current state: Under Discussion
Discussion thread: HERE
JIRA: KAFKA-4133
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation
With KIP-74, we now have a good way to limit the size of fetch responses, but it may still be difficult for users to control overall memory since the consumer will send fetches in parallel to all the brokers which own partitions that it is subscribed to. To give users finer control, it might make sense to add a `max.inflight.fetches
` setting to limit the total number of concurrent fetches at any time.
Public Interfaces
The following option will be added for consumers to configure (in ConsumerConfigs.java)
:
max.inflight.fetches
(Integer) Specifies the maximum number of in-flight fetches the consumer can make. Defaults toInteger.MAX_VALUE
.
Proposed Changes
Update Fetcher.java
to check the number of existing in-flight fetches (this is already tracked by numInFlightFetches)
before initiating new fetch requests in createFetches().
Compatibility, Deprecation, and Migration Plan
This KIP should be transparent to users not interested in setting this new configuration. Users wanting to take advantage of this new feature will just need to add this new settings to their consumer's properties.
Rejected Alternatives
Use bounded memory pool to control memory usage (KIP-72):
While this might be the best way to handle that on the server side it's unclear if this would suit the client well. Usually the client has a rough idea about how many partitions it will be subscribed to so it's easier to size the maximum number of concurrent fetches.