You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 6 Next »

Status

Current state: Draft [One of "Under Discussion", "Accepted", "Rejected"]

Discussion thread: not yet

JIRA: KAFKA-4133

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

With KIP-74, we now have a good way to limit the size of fetch responses, but it may still be difficult for users to control overall memory since the consumer will send fetches in parallel to all the brokers which own partitions that it is subscribed to. To give users finer control, it might make sense to add a `max.in.flight.fetches` setting to limit the total number of concurrent fetches at any time.

Public Interfaces

The following option will be added for consumers to configure (in ConsumerConfigs.java):

  1. max.in.flight.fetches (Integer) Specifies the maximum number of in-flight fetches the consumer can make. Defaults to Integer.MAX_VALUE.

Proposed Changes

Update Fetcher.java to take into account the number of existing in-flight fetches before initiating a new fetch request. The total number of in-flight requests is already tracked, so we can retrieve how many fetch are in-flight and take this value into account when creating new fetches in Fetcher.createFetchRequests().

Compatibility, Deprecation, and Migration Plan

This KIP should be transparent to users not interested in setting this new configuration.

Rejected Alternatives

None so far

  • No labels