...

Quota management via Admin Client has gone through a couple drafts of proposals (KIP-248, KIP-422). While improvements have been made to the Admin interface for configuration handling, fitting quotas into the API is awkward as they don't fit the natural key-value pairing, nor is the configuration output expressive enough to return all useful information. Therefore, it'd be beneficial to have a quota-native API for managing quotas, which would offer an intuitive and less error-prone interface, convey additional information beyond what the configuration APIs provide, and enable for future extensibility as quotas types are added or evolved.

Background

Quotas By default, quotas are currently defined in terms of a user and client ID, where the user acts as a specific an opaque principal name, and the client ID as a more generic group identifier. When setting quotas, an administrator has flexibility in how it specifies the user and client ID for which the quota applies to, where the user and client ID may be specifically named, indicated as the a default, or omitted entirely. Since there's multiple ways to specify quotas, a hierarchy structure for which quotas apply must be appliedquotas have flexible configurations, there is a method for resolving the quotas that apply to a request: a hierarchy structure is used, where the most specific , defined quota will be matched to a request's user and client ID.

...

As such, reasoning around quotas can be complex, as it's not immediately obvious which quotas may apply to a given user and/or client ID. Providing descriptive information about how quotas are matched is the first goal of this KIP. Likewise, retrieving and modifying quota values can be done in a more expressive and robust way, which is the second goal of this the KIP.

APIs

In order to clearly specify the APIs, let's first disambiguate some terminologyNote: Every client request that is processed is associated with a quota entity, which is a map of entity types to their corresponding entity names for the request. Using the current entity types in the previous examples, an entity may be is of the form {user=test-user, clientId client-id=test-client}. When , where user and client-id are the types, and test-user and test-client are the names. However, when specifying a quota configuration entry, only a subset of the entity types need to be provided, which is referred to as an entity match, for example, {user=<default>}.

APIs

For describing quotas, there's two modes of operation that are necessary for administration: config-centric, and entity-centric.

1. The config-centric mode describes what is exactly specified for the configuration for the given entity match. However, it would also be useful to also be able to determine which matches have configuration values defined, so the presence of a filter is used for gathering information about the entity matches that the administrator is interested in. This is DescribeQuotas.
2. The entity-centric mode describes what quotas are applied apply to an entity. Note that an entity may match to various configuration entries depending on how the quotas are specified, e.g. the producer byte rate may be specified for the user, but the consumer byte rate for the client ID. Since upon querying this information, it may not be clear how quotas were matched for an entity from the configuration, additional information should be returned to give provide more context. This is DescribeEffectiveQuotas.

Altering quotas only works on a config-centric manner, and therefore doesn't need distinguishing. For a given entity match, the administrator should be able to specify which quotas apply, or alternatively remove existing quotas so they don't no longer match. This is AlterQuotas.

...

This KIP introduces the concept of a quota unit to be applied to a quota value. Currently, only a single unit is used for quotas: bytes-per-second, however this has limitations to effective quota management. For example, since it's a global throughput value, it doesn't scale well as brokers are added or removed, and so a broker-bytes-per-second unit could be added to better manage this behavior. As units are added, the possible quota configuration entries becomes the cross product of the quota types by the quota units, which means that it'd be possible to specify bytes-per-second both on a global and per-broker basis, and quota enforcement would occur at whichever limit was hit first. Additional considerations could be made for a fair-share system. Units , where units of shares could be configured to entitiesfor quotas, and when bandwidth is contested, the share count of the various active entities could be used to determine their restricted throughput.

While itIt's beyond the scope of this KIP to implement such add new units and implement their corresponding functionality in the broker, however it must be noted for future extensibility of the APIs.

...

While there's two defined entity types in AK, a server-side plugin mechanism allows for further expansion. Likewise, as use cases evolve, finer-grained quota control may be necessary. Therefore, entity types should not be statically bound to what's publicly defined constants, and instead the API should support flexible entity types by interpreting them as a String identifier. Any entity types that the broker doesn't understand should return throw an error IllegalArgumentException back to the client.

The quota types (producer byte rate, consumer byte rate, etc.) and units should also be given the same consideration. The possible quota applications may expand in the future, and the API shouldn't lock in which quota types are accessible. Modification of quota types/units that are unknown should also fail with error.

For forming entities, since Since a fixed set of entity types aren't defined, a an entity should be represented by a Map<String, String> should be used to map , which maps an entity type to the entity name.

Public Interfaces

...

/**
 * Describes a fully-qualified entity.
 */
public class QuotaEntity {
    /**
     * Type of an entity entry.
     */
    public enum Type {
        USER,
        CLIENT_ID;
    }

    /**
     * Represents the default name for an entity, i.e. the entity that's matched
     * when an exact match isn't found.
     */
    public final static String QUOTA_ENTITY_NAME_DEFAULT = // implementation defined

    /**
     * `entries` describes the fully-qualified entity. The key is a {@code Type} string, however
     * there may also exist keys that are not enumerated by {@code Type} that still apply, e.g.
     * the server may internally associate another type. ItWhen querying entities, it's necessary
     * to return all quota types
     * because quota values for these types may influence the effective
     * quota value. However,
 when    * altering a quota, any types that aren't specified shouldmust be able
     * to be inferred by the
     * server, otherwise an error is returned.
     *
     * For example, {("CLIENT_ID" -> "test-client"),
     *               ("USER" -> "test-user"),
     *               ("GROUP" -> "internal-group")}.
     */
    public QuotaEntity(Map<String, String> entries);
}

/**
 * Describes a quota key.
 */
public class QuotaKey {
    /**
     * The quota types.
     */
    public enum Type {
        CONSUMER_BYTE_RATE,
        PRODUCER_BYTE_RATE,
        REQUEST_PERCENTAGE;
    }

    /**
     * The units for a quota value. Note there may be multiple units for a given quota type
     * that influences quota behavior.
     */
    public enum Units {
        RATE_BPS;
    }

    /**
     * @param type the quota type
     * @param units the units for the quota type
     */
    public QuotaKey(Type type, Units units);
}

/**
 * Describes a quota entity filter.
 */
public class QuotaFilter {
    public enum Rule {
        EXACT,    // exact name match
        PREFIX;   // matches all names with the given prefix
    }

    /**
     * A filtering rule to be applied.
     *
     * @param entityType the entity type the rule applies to
     * @param rule the rule to apply
     * @param match the non-null string that's applied by the rule
     */
    public QuotaFilter(QuotaEntity.Type entityType, Rule rule, String match);
}

...