Status

Current state: Under Discussion

Discussion thread: Here

JIRA: Unable to render Jira issues macro, execution error.

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Consumer groups are an essential mechanism of Kafka. They allow consumers to share load and elastically scale by dynamically assigning the partitions of topics to consumers. In our current model of consumer groups, whenever a rebalance happens every consumer from that group experiences downtime - their poll() calls block until every other consumer in the group calls poll(). That is due to the fact that every consumer needs to call JoinGroup in a rebalance scenario in order to confirm it is still in the group. 

Today, if the client has configured `max.poll.interval.ms` to a large value, the group coordinator broker will take in an unlimited number of join group requests and the rebalance could therefore continue for an unbounded amount of time. There is also the
potential risk described in Unable to render Jira issues macro, execution error.  where N faulty (or even malicious) clients could result in the broker thinking more than N consumers are joining during the rebalance. (This has the potential to burst broker memory before the session timeout occurs and puts additional CPU strain).  
The root of the problem isn't necessarily the client's behavior (clients can behave any way they want), it is the fact that the broker has no way to shield itself from such a scenario.

Further, large consumer groups are not very practical with our current model due to two reasons:
1. The more consumers there are, the likelier it is that one will fail/timeout its session, causing a rebalance
2. Rebalances are upper-bounded in time by the slowest-reacting consumer. The more consumers, the higher the chance one is slow (e.g called poll() right before the rebalance and is busy processing the records offline). This means that rebalances are more likely to be long-lived and disruptive to consumer applications


To ensure stability of the broker, this KIP proposes the addition of a configurable upper-bound for the number of consumers in a consumer group. Adding such a config with a sensible default value and documentation would ensure broker protection and help guide users on using consumer groups effectively.

Public Interfaces

Add a new cluster-level group.max.size config with a default value of 250.

Add a new response error:

GROUP_MAX_SIZE_REACHED(77, "Consumer group is already at its full capacity.",
 GroupMaxSizeReachedException::new);

Proposed Changes

We shall block registration of new member once a group reaches its configured capacity. Any subsequent `JoinGroup` requests will receive a response with the `GROUP_MAX_SIZE_REACHED` error.

Since the cap should never be reached in practice, the consumer will fatally exit upon receiving this error message.

Compatibility, Deprecation, and Migration Plan

This is a backward compatible change. Old clients will still fail by converting the new error to the non-retriable UnknownServerException

Rejected Alternatives

• Topic-level config
  • It is harder to enforce since a consumer group may touch multiple topics. One approach would be to take the min/max of every topic's group size configuration.
  • This fine-grained configurability does not seem needed for the time being and may best be left for the future if the need arises
• There are other ways of limiting how long a rebalance can take, discussed here
  • In the form of time - have a max rebalance timeout (decoupled from `max.poll.interval.ms`)
    • Lack strictness, a sufficiently buggy/malicious client could still overload the broker in a small time period
  • In the form of memory - have a maximum memory bound that can be taken up by a single group
    • Lacks intuitiveness, users shouldn't think about how much memory a consumer group is taking