Page History

...

Since the broker has to accept the connection in order to know the IP address, limiting connection creation rate per IP means that the enforcement of the limit has to happen after connection is accepted. Dropping the connection right away as broker does for per-IP connection count limits (KIP-402) does not work as well for connection rate limiting, because the offending client is likely to immediately reconnect. At the same time, connection rate limiting by only delaying processing of connections would not work for cases where the connection creation rate is continuously higher than the set limits, which would create a large backlog. This case could be more common for clients that come through a proxy, and where there could be potentially a large number of incoming connections with the same IP. To address these two issues, our approach is as follows. If connection creation rate is reached for a specific IP address, the broker will delay processing the connection by an amount of time that brings the rate within the limit or 1 second, whichever is earliest. After the delay, if the per-IP quota is still violated, the connection will be cleaned up; otherwise, the connection will be accepted.

Metrics

New metrics that track broker-wide or per-listener rate of accepting connections, and are used by SocketServer to detect connection acceptance rate quota violation:

•  kafka.network:type=socket-server-metrics,name=connection-accept-rate,listener={listenerName}

  • Type: Rate
  • Description: Rate of connections accepted per second on a given listener

•  kafka.network:type=socket-server-metrics,name=broker-connection-accept-rate

  • Type: Rate
  • Description: Broker-wide rate of connections accepted per second

New metrics that track average throttle time of accepting a new connection due to reaching connection acceptance rate limit:

• kafka.network:type=socket-server-metrics,name=connection-accept-throttle-time,listener={listenerName}

  • Type: SampledStat.Avg
  • Description: Average throttle time due to violating per-listener or broker-wide connection acceptance rate quota on a given listener.

No new metrics will be added. The existing metric (kafka.network:type=Acceptor,name=AcceptorBlockedPercent,listener={listenerName}) that tracks the amount of time Acceptor is blocked from accepting connections will now additionally include the amount of time Acceptor is blocked due to hitting connection create limit (in addition to the time blocked due to hitting the maximum limit on currently active connections). 

...