Status

Current state: Draft

Discussion thread: TODO

JIRA: TODO

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Quota management via Admin Client has gone through a couple drafts of proposals (KIP-248, KIP-422). While improvements have been made to the Admin interface for configuration handling, fitting quotas into the API would be restrictive because they don't fit the natural key-value pairing. Therefore, it'd be beneficial to have a quota-native API for administrating quotas, which would offer a more direct and less error-prone interface, convey additional useful information beyond what a configuration could provide, and provide for future extensibility as quotas types are added or evolve.

Background

Quotas are defined in terms of a user and client ID, where the user acts as a specific opaque principal, and the client ID as a more generic group identifier.

When setting quotas, an administrator has flexibility in how it specifies the user and client ID for which the quota applies to, where the user and client ID may be named, indicated as the default, or omitted entirely. This creates a hierarchy structure for which quotas apply, where the most specific, defined quota will be matched to a request's user and client ID.

As represented by the current ZK node structure, the order in which quotas are matched are as follows. Note <user> is a specified user name, <client-id> is a specified client ID, and <default> is a special default user/client ID represented as the literal string "<default>".

        /config/users/<user>/clients/<client-id>
        /config/users/<user>/clients/<default>
        /config/users/<user>
        /config/users/<default>/clients/<client-id>
        /config/users/<default>/clients/<default>
        /config/users/<default>
        /config/clients/<client-id>
        /config/clients/<default>

As such, reasoning around quotas can be complex, as it's not immediately obvious which quotas may apply to a given user and/or client ID. Providing descriptive information is a goal of this KIP.

Public Interfaces

class QuotaFilter {
    public static final String DEFAULT_USER;
    public static final String DEFAULT_CLIENT_ID;
    public static final String ANY_USER;
    public static final String ANY_CLIENT_ID;

    String user;
    String clientId;
}

/**
 * The quota types.
 */
public enum QuotaType {
    CONSUMER_BYTE_RATE,
    PRODUCER_BYTE_RATE,
    REQUEST_PERCENTAGE,
}

/**
 * The quota configuration for a specific user and client ID.
 */
public class QuotaConfig {

    /**
     * Where in the hierarchy the quota value was specified.
     */
    public enum Source {
        USER_CLIENT,
        USER_DEFAULT_CLIENT,
        USER,
        DEFAULT_USER_CLIENT,
        DEFAULT_USER_DEFAULT_CLIENT,
        DEFAULT_USER,
        CLIENT,
        DEFAULT_CLIENT,
        DEFAULT,
    }

    /**
     * Information about a quota value.
     */
    public class Value {
        // The source for the value.
        private final Source source;

        // The non-null quota's value.
        private final Double value;

        // Basic constructor/getters.
        public Value(Source source, Double value);
        public Source source();
        public Double value();
    }

    /**
     * Information about the value for a quota type.
     */
    public class Entry {
        // The active quota value.
        private final Value value;

        // List of all values lower that are ignored due to being lower in the hierarchy.
        private final List<Value> overriddenValues;

        // Basic constructor/getters.
        public Entry(Value value, List<Value> overriddenValues);
        public Value value();
        public List<Value> overriddenValues();
    }

    /**
     * Maps quota type to its configuration entry.
     *
     * Note that, depending upon the applied quota filter, this config may not map every
     * quota type to an entry. If a key is not contained in the map, then that quota type's value
     * is not specified.
     */
    public Map<QuotaType, Entry> config();
}

/**
 * Identifies a quota entity for a given user and client ID. If the provided user or client ID is
 * `Optional.empty()`, then the default name is used, otherwise if null, then the name is omitted
 * entirely. At least one of user or client ID must be specified.
 *
 * For example:
 *
 * {user="kafka-user", clientId="kafka-client-id"} corresponds to quota source `USER_CLIENT` with
 * specified user "kafka-user" and client ID "kafka-client-id".
 *
 *  {user="kafka-user", clientId=Optional.empty()} corresponds to quota source `USER_DEFAULT_CLIENT`
 * with specified user "kafka-user" and default client ID.
 *
 * {user=null, clientId="kafka-client-id"`} corresponds to quota source `CLIENT` for client with
 * ID "kafka-client-id".
 */
class QuotaEntity {
    QuotaEntity(Optional<String> user, Optional<String> clientId);
}

class DescribeQuotasOptions {
    // Whether to include the overridden values for every quota type in the result.
    boolean wantOverriddenValues;
}

class DescribeQuotasResult {
    Map<QuotaEntity, QuotaConfig> configs;
}

/**
 * Describes the quotas for the provided filters.
 */
DescribeQuotasResult describeQuotas(Collection<QuotaFilter> quotaFilters);

class QuotaAlteration {
    // The quota type to alter.
    QuotaType type;

    // The new value for the quota, otherwise if null, then the existing value is cleared.
    Double value;
}

class AlterQuotasOptions {
    // Default.
}

class AlterQuotasResult {
    // Maps a quota entity update to its future.
    private final Map<QuotaEntity, KafkaFuture<Void>> futures;
}

/**
 * Alters the quotas as specified for the provided quota entities.
 *
 * @param alterations the alterations to perform
 *  @return the result of the alterations
 */
AlterQuotasResult alterQuotas(Map<QuotaEntity, QuotaAlteration> alterations);

Proposed Changes

Compatibility, Deprecation, and Migration Plan

All changes would be forward-compatible, and no migration plan is necessary. It's outside the scope of this KIP to deprecate any functionality.

Rejected Alternatives