Table of Contents |
---|
Status
Current state: Under Discussion Accepted
Discussion thread: here
JIRA: here
...
The client left as is risks imposing heavy load on ZooKeeper. ZooKeeper itself only has a coarse-grained throttling mechanism in place through its zookeeper.globalOutstandingLimit
config which defaults to 1000. This config is insufficient for several reasons:1.
- the limit is meant to protect ZooKeeper from memory pressure associated with a backlog of requests.
...
- the limit is applied across all connections. Even with this config, one misbehaved client will affect the other clients.
We need a throttling mechanism in the client-side to give administrators control over Kafka's impact on ZooKeeper, and for this we propose a new broker config called zookeeper.max.in.flight.requests
.
...
This config must be set to at least one1.
The default value is set to 10. We ran experiments showing the impact of zookeeper.max.in.flight.requests
on completion times for various ZooKeeper-intensive controller protocols. The default was chosen to be the smallest number beyond which the experiment results have found diminishing returns.
...
- Set the default value to be unbounded (Integer.MAX_VALUE), effectively stating that no such limit be applied and to pipeline as aggressively as possible. This was rejected because a config should not default to removing control over the system.
- Set the default value to one1, effectively disabling pipelining and maintaining Kafka's synchronous requests to ZooKeeper. This was rejected because we want users to see the benefits of pipelining.