...
If per-IP or default IP quota is not configured, we will use Int.MaxValue as a default to ensure backward compatibility.
We will use the AlterClientQuota
and DescribeClientQuota
APIs to manage IP connection rate quotas. Adding support for IP quotas will not require any changes to the protocol, since entity type and name are represented flexibly as Strings. For ease of use, we will define a new recognized IP entity type in org.apache.kafka.common.quota.ClientQuotaEntity:
Code Block |
---|
/**
* Describes a client quota entity, which is a mapping of entity types to their names.
*/
public class ClientQuotaEntity {
/**
* The type of an entity entry.
*/
public static final String USER = "user";
public static final String CLIENT_ID = "client-id";
public static final String IP = "IP";
} |
Since IP quotas are orthogonal with user/client ID quotas the following is to disambiguate how the new IP
entity will interact with the existing client-id
and user
entities.
A DescribeClientQuotasRequest
constructed with a filter for both ip entities and either of user or client-id represents a logical AND
of IP quotas and user/client quotas. However, since there are no quotas that satisfy both IP and either of user/client-id, such a request is likely due to user error. In such a case we will return an INVALID_REQUEST
error.
Likewise, an AlterClientQuotasRequest
with both an IP entity and either of user or client-id entity will return an INVALID_REQUEST
error, as there are no shared quota properties between the two entity types.
Tools
kafka-configs.sh
will be extended to support per-IP connection rate quotas. A new entity type IPs
will be added with the following key (and a value of type Int):
...
kafka.network:type=socket-server-metrics,name=connection-accept-rate,listener={listenerName}
- Type: Rate
- Description: Rate of connections accepted per second on a given listener
kafka.network:type=socket-server-metrics,name=broker-connection-accept-rate
- Type: Rate
- Description: Broker-wide rate of connections accepted per second
kafka.network:type=socket-server-metrics,name=connection-accept-rate,ip={ipAddress}
- Type: Rate
- Description: Rate of connections accepted per second for a given IP
New metrics that track average throttle time of accepting a new connection due to reaching connection acceptance rate limit:
kafka.network:type=socket-server-metrics,name=connection-accept-throttle-time,listener={listenerName}
- Type: SampledStat.Avg
- Description: Average throttle time due to violating per-listener or broker-wide connection acceptance rate quota on a given listener.
kafka.network:type=socket-server-metrics,name=ip-connection-accept-throttle-time,listener={listenerName}
- Type: SampledStat.Avg
- Description: Average throttle time due to violating ip connection rate quota on a given listener. IP throttling is reported at a listener granularity rather than at an IP granularity to keep a smaller metrics overhead.
The existing metric (kafka.network:type=Acceptor,name=AcceptorBlockedPercent,listener={listenerName}) that tracks the amount of time Acceptor
is blocked from accepting connections will now additionally include the amount of time Acceptor
is blocked due to hitting connection create limit (in addition to the time blocked due to hitting the maximum limit on currently active connections).
...
Broker configurations to set limits on per IP connection attempt rates
Add broker configurations for setting the default per-IP connection attempt limit and per-IP limit overrides:
Config option: Name: max.connection.creation.rate.per.ip
Type: Int
Default value: Int.MaxValue
Config option: Name: max.connection.creation.rate.per.ip.overrides
Type: String
Default value: ""
We chose the option of making per-IP quota similar to other quotas, configured as a dynamic quota for entity type "ip" as well as defaults.
Use incrementalAlterConfigs/describeConfigs for IP quotas
DescribeConfigsRequest
requires the user to specify a list of ConfigResource
which consists of a Type
and Name
for which to retrieve the desired configs. An empty name indicates that we want to describe the configs for the entity type default. This works well for describing broker and topic configs, because the full list of brokers and topics is retrievable via other requests.
However, for IPs there is no mechanism for IP discovery, and DescribeConfigsRequest
has no way to indicate that a request should list all configs for an entity type without also giving a list of entity names. This would make discovery of which IPs have a connection rate quota unfeasible, so this was rejected.