You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 30 Next »

Status

Current state: Draft

Discussion thread: TODO

JIRA: TODO

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Quota management via Admin Client has gone through a couple drafts of proposals (KIP-248, KIP-422). While improvements have been made to the Admin interface for configuration handling, fitting quotas into the API would be restrictive because they don't fit the natural key-value pairing. Therefore, it'd be beneficial to have a quota-native API for administrating quotas, which would offer an intuitive and less error-prone interface, convey additional useful information beyond what the configuration APIs could provide, and provide for future extensibility as quotas types are added or evolved.

Background

Quotas are defined in terms of a user and client ID, where the user acts as a specific opaque principal, and the client ID as a more generic group identifier.

When setting quotas, an administrator has flexibility in how it specifies the user and client ID for which the quota applies to, where the user and client ID may be named, indicated as the default, or omitted entirely. This creates a hierarchy structure for which quotas apply, where the most specific, defined quota will be matched to a request's user and client ID.

As represented by the current ZK node structure, the order in which quotas are matched are as follows. Note <user> is a specified user name, <client-id> is a specified client ID, and <default> is a special default user/client ID represented as the literal string "<default>".

        /config/users/<user>/clients/<client-id>
        /config/users/<user>/clients/<default>
        /config/users/<user>
        /config/users/<default>/clients/<client-id>
        /config/users/<default>/clients/<default>
        /config/users/<default>
        /config/clients/<client-id>
        /config/clients/<default>

As such, reasoning around quotas can be complex, as it's not immediately obvious which quotas may apply to a given user and/or client ID. Providing descriptive information is a goal of this KIP.

Public Interfaces


Common types in package org.apache.kafka.common.quota:

/**
 * Describes a fully-qualified entity.
 */
public class QuotaEntity {
    /**
     * Type of an entity entry.
     */
    public enum Type {
        USER,
        CLIENT_ID;
    }

    /**
     * Represents the default name for an entity, i.e. the entity that's matched
     * when an exact match isn't found.
     */
    public final static String QUOTA_ENTITY_NAME_DEFAULT = // implementation defined

    /**
     * `entries` describes the fully-qualified entity. The key is a {@code Type} string, however
     * there may also exist keys that are not enumerated by {@code Type} that still apply, e.g.
     * the server may internally associate another type. It's necessary to return all quota types
     * because quota values for these types may influence the effective quota value. However,
     * altering a quota, any types that aren't specified should be able to be inferred by the
     * server, otherwise an error is returned.
     *
     * For example, {("CLIENT_ID" -> "test-client"),
     *               ("USER" -> "test-user"),
     *               ("GROUP" -> "internal-group")}.
     */
    public QuotaEntity(Map<String, String> entries);
}

/**
 * Describes a quota key.
 */
public class QuotaKey {
    /**
     * The quota types.
     */
    public enum Type {
        CONSUMER_BYTE_RATE,
        PRODUCER_BYTE_RATE,
        REQUEST_PERCENTAGE;
    }

    /**
     * The units for a quota value. Note there may be multiple units for a given quota type
     * that influences quota behavior.
     */
    public enum Units {
        RATE_BPS;
    }

    /**
     * @param type the quota type
     * @param units the units for the quota type
     */
    public QuotaKey(Type type, Units units);
}

/**
 * Describes a quota entity filter.
 */
public class QuotaFilter {
    public enum Rule {
        EXACT,    // exact name match
        PREFIX;   // matches all names with the given prefix
    }

    /**
     * A filtering rule to be applied.
     *
     * @param entityType the entity type the rule applies to
     * @param rule the rule to apply
     * @param match the non-null string that's applied by the rule
     */
    public QuotaFilter(QuotaEntity.Type entityType, Rule rule, String match);
}

DescribeQuotas API:

public class DescribeQuotasOptions extends AbstractOptions<DescribeQuotasOptions> {
    // Empty.
}

/**
 * The result of the {@link Admin#describeQuotas(Collection<QuotaFilter>, DescribeQuotasOptions)} call.
 */
public class DescribeQuotasResult {

    /**
     * Maps an entity to its configured quota value(s). Note if no value is defined for a quota
     * type for that entity's config, then it is not included in the resulting value map.
     *
     * @param entities the collection of entities that matched the filter
     */
    DescribeQuotasResult(KafkaFuture<Map<QuotaEntity, Map<QuotaKey, Long>>> entities);

    /**
     * Returns a map from quota entity to a future which can be used to check the status of the operation.
     */
    public KafkaFuture<Map<QuotaEntity, Map<QuotaKey, Long>>> entities();
}

public interface Admin extends AutoCloseable {
    ...

    /**
     * Describes all entities matching all provided filters (logical AND) that have at least one
     * quota value defined.
     *
     * @param filters filtering rules to apply to matching entities
     * @param options the options to use
     * @return result containing all matching entities
     */
    DescribeQuotasResult describeQuotas(Collection<QuotaFilter> filters, DescribeQuotasOptions options);
}

DescribeEffectiveQuotas

public class DescribeEffectiveQuotasOptions extends AbstractOptions<DescribeEffectiveQuotasOptions> {

    /**
     * Whether to exclude the list of overridden values for every quota type in the result.
     */
    DescribeEffectiveQuotasOptions setOmitOverriddenValues(boolean omitOverriddenValues);
}

/**
 * The result of the {@link Admin#describeEffectiveQuotas(Collection<QuotaEntity>, DescribeEffectiveQuotasOptions)} call.
 */
public class DescribeEffectiveQuotasResult {
    /**
     * Information about a specific quota configuration entry.
     */
    public class Entry {
        /**
         * @param source the entity source for the value
         * @param value the non-null value
         */
        public Entry(QuotaEntity source, Long value);
    }

    /**
     * Information about the value for a quota type.
     */
    public class Value {
        /**
         * @param entry the quota entry
         * @param overriddenEntries all values that are overridden due to being lower in
         *                          specificity, or null if not requested
         */
        public Value(Entry entry, List<Entry> overriddenEntries);
    }

    /**
     * Maps a collection of entities to their effective quota values.
     *
     * @param config the quota configuration for the requested entities
     */
    public DescribeEffectiveQuotasResult(Map<QuotaEntity, KafkaFuture<Map<QuotaKey, Value>>> config);

    /**
     * Returns a map from quota entity to a future which can be used to check the status of the operation.
     */
    public Map<QuotaEntity, KafkaFuture<Map<QuotaKey, Value>>> config();

    /**
     * Returns a future which succeeds only if all quota descriptions succeed.
     */
    public KafkaFuture<Void> all();
}

public interface Admin extends AutoCloseable {
    ...

    /**
     * Describes the effective quotas for the provided entities.
     *
     * @param entities the entities to describe the effective quotas for
     * @param options the options to use
     * @return the effective quotas for the entities
     */
    DescribeEffectiveQuotasResult describeEffectiveQuotas(Collection<QuotaEntity> entities, DescribeEffectiveQuotasOptions options);
}

AlterQuotas

public class AlterQuotasEntry {
    public class Op {
        /**
         * @param key the quota type and units to alter
         * @param value if set then the existing value is updated,
         *              otherwise if null, the existing value is cleared
         */
        public Op(QuotaKey key, Long value);
    }

    /**
     * @param entity the entity whose config will be modified
     * @param ops the alteration to perform - if value is set, then the existing value is updated,
     *            otherwise if null, the existing value is cleared
     */
    public AlterQuotasEntry(QuotaEntity entity, Collection<Op> ops);
}

public class AlterQuotasOptions extends AbstractOptions<AlterQuotasOptions> {
    /**
     * Sets whether the request should be validated without altering the configs.
     */
    public AlterQuotasOptions validateOnly(boolean validateOnly);
}

/**
 * The result of the {@link Admin#alterQuotas(Collection<AlterQuotasEntry>, AlterQuotasOptions)} call.
 *
 * The API of this class is evolving, see {@link Admin} for details.
 */
public class AlterQuotasResult {
    public AlterQuotasResult(Map<QuotaEntity, KafkaFuture<Void>> futures);

    /**
     * Returns a map from quota entity to a future which can be used to check the status of the operation.
     */
    public Map<QuotaEntity, KafkaFuture<Void>> values();

    /**
     * Returns a future which succeeds only if all quota alterations succeed.
     */
    public KafkaFuture<Void> all();
}

public interface Admin extends AutoCloseable {
    ...

    /**
     * Alters the quotas as specified for the entries.
     *
     * @param alterations the alterations to perform
     * @return the result of the alterations
     */
    AlterQuotasResult alterQuotas(Collection<AlterQuotasEntry> entries, AlterQuotasOptions options);
}


Proposed Changes


DescribeQuotas:

{
  "apiKey": 48,
  "type": "request",
  "name": "DescribeQuotasRequest",
  "validVersions": "0",
  "flexibleVersions": "none",
  "fields": [
    { "name": "Filter", "type": "[]QuotaFilterData", "versions": "0+",
      "about": "Filters to apply to quota entities.", "fields": [
      { "name": "EntityType", "type": "string", "versions": "0+",
        "about": "The entity type that the filter applies to." },
      { "name": "Rule", "type": "string", "versions": "0+",
        "about": "The rule the filter performs." },
      { "name": "Match", "type": "string", "versions": "0+",
        "about": "The string to apply the rule against." }
    ]}
  ]
}

{
  "apiKey": 48,
  "type": "response",
  "name": "DescribeQuotasResponse",
  "validVersions": "0",
  "flexibleVersions": "none",
  "fields": [
    { "name": "ThrottleTimeMs", "type": "int32", "versions": "0+",
      "about": "The duration in milliseconds for which the request was throttled due to a quota violation, or zero if the request did not violate any quota." },
    { "name": "Entry", "type": "[]EntryData", "versions": "0+",
      "about": "A result entry.", "fields": [
      { "name": "ErrorCode", "type": "int16", "versions": "0+",
        "about": "The error code, or `0` if the quota description succeeded." },
      { "name": "ErrorMessage", "type": "string", "versions": "0+", "nullableVersions": "0+",
        "about": "The error message, or `null` if the quota description succeeded." },
      { "name": "Entity", "type": "[]QuotaEntityData", "versions": "0+",
        "about": "The quota entity description.", "fields": [
        { "name": "EntityType", "type": "string", "versions": "0+",
          "about": "The entity type." },
        { "name": "EntityName", "type": "string", "versions": "0+",
          "about": "The entity name." }
      ]},
      { "name": "Type", "type": "string", "versions": "0+",
        "about": "The quota type." },
      { "name": "Units", "type": "string", "versions": "0+",
        "about": "The units for the value." },
      { "name": "Value", "type": "int64", "versions": "0+",
        "about": "The quota value." }
    ]}
  ]
}

DescribeEffectiveQuotas:

{
  "apiKey": 49,
  "type": "request",
  "name": "DescribeEffectiveQuotasRequest",
  "validVersions": "0",
  "flexibleVersions": "none",
  "fields": [
    { "name": "Entity", "type": "[]QuotaEntityData", "versions": "0+",
      "about": "The quota entity description.", "fields": [
      { "name": "EntityType", "type": "string", "versions": "0+",
        "about": "The entity type." },
      { "name": "EntityName", "type": "string", "versions": "0+",
        "about": "The entity name." }
    ]},
    { "name": "OmitOverriddenValues", "type": "bool", "versions": "0+",
      "about": "Whether to exclude the list of overridden values for every quota type." }
  ]
}

{
  "apiKey": 49,
  "type": "response",
  "name": "DescribeEffectiveQuotasResponse",
  "validVersions": "0",
  "flexibleVersions": "none",
  "fields": [
    { "name": "ThrottleTimeMs", "type": "int32", "versions": "0+",
      "about": "The duration in milliseconds for which the request was throttled due to a quota violation, or zero if the request did not violate any quota." },
    { "name": "Entry", "type": "[]QuotaEntryData", "versions": "0+",
      "about": "Effective quota entries.", "fields": [
      { "name": "ErrorCode", "type": "int16", "versions": "0+",
        "about": "The error code, or `0` if the effective quota description succeeded." },
      { "name": "ErrorMessage", "type": "string", "versions": "0+", "nullableVersions": "0+",
        "about": "The error message, or `null` if the effective quota description succeeded." },
      { "name": "QuotaEntity", "type": "[]QuotaEntity", "versions": "0+",
        "about": "Effective quota entries.", "fields": [
        { "name": "EntityType", "type": "string", "versions": "0+",
          "about": "The entity type." },
        { "name": "EntityName", "type": "string", "versions": "0+",
          "about": "The entity name." }
      ]},
      { "name": "QuotaValues", "type": "[]QuotaValueData", "versions": "0+",
        "about": "Quota configuration values.", "fields": [
        { "name": "Type", "type": "string", "versions": "0+",
          "about": "The quota type." },
        { "name": "Units", "type": "string", "versions": "0+",
          "about": "The units for the quota type." },
        { "name": "Entry", "type": "[]ValueEntryData", "versions": "0+",
          "about": "Quota value entries.", "fields": [
          { "name": "QuotaEntity", "type": "[]ValueQuotaEntity", "versions": "0+",
            "about": "Effective quota entries.", "fields": [
            { "name": "EntityType", "type": "string", "versions": "0+",
              "about": "The entity type." },
            { "name": "EntityName", "type": "string", "versions": "0+",
              "about": "The entity name." }
          ]},
          { "name": "Value", "type": "int64", "versions": "0+",
            "about": "The quota configuration value." }
        ]}
      ]}
    ]}
  ]
}


AlterQuotas:

{
  "apiKey": 50,
  "type": "request",
  "name": "AlterQuotasRequest",
  "validVersions": "0",
  "flexibleVersions": "none",
  "fields": [
    { "name": "Entry", "type": "[]EntryData", "versions": "0+",
      "about": "The quota configuration entries to alter.", "fields": [
      { "name": "QuotaEntity", "type": "[]QuotaEntity", "versions": "0+",
        "about": "The quota entity to alter.", "fields": [
        { "name": "EntityType", "type": "string", "versions": "0+",
          "about": "The entity type." },
        { "name": "EntityName", "type": "string", "versions": "0+",
          "about": "The name of the entity." }
      ]},
      { "name": "Op", "type": "[]OpData", "versions": "0+",
        "about": "An individual quota configuration entry to alter.", "fields": [
        { "name": "Type", "type": "string", "versions": "0+",
          "about": "The quota type." },
        { "name": "Units", "type": "string", "versions": "0+",
          "about": "The units for the quota type." },
        { "name": "Value", "type": "int64", "versions": "0+",
          "about": "The value to set, otherwise ignored if the value is to be removed." },
        { "name": "Remove", "type": "bool", "versions": "0+",
          "about": "Whether the quota configuration value should be removed, otherwise set." }
      ]}
    ]},
    { "name": "ValidateOnly", "type": "bool", "versions": "0+",
      "about": "Whether the alteration should be validated, but not performed." }
  ]
}

{
  "apiKey": 50,
  "type": "response",
  "name": "AlterQuotasResponse",
  "validVersions": "0",
  "flexibleVersions": "none",
  "fields": [
    { "name": "ThrottleTimeMs", "type": "int32", "versions": "0+",
      "about": "The duration in milliseconds for which the request was throttled due to a quota violation, or zero if the request did not violate any quota." },
    { "name": "Entry", "type": "[]EntryData", "versions": "0+",
      "about": "The quota configuration entries to alter.", "fields": [
      { "name": "ErrorCode", "type": "int16", "versions": "0+",
        "about": "The error code, or `0` if the quota alteration succeeded." },
      { "name": "ErrorMessage", "type": "string", "versions": "0+", "nullableVersions": "0+",
        "about": "The error message, or `null` if the quota alteration succeeded." },
      { "name": "QuotaEntity", "type": "[]QuotaEntity", "versions": "0+",
        "about": "The quota entity to alter.", "fields": [
        { "name": "EntityType", "type": "string", "versions": "0+",
          "about": "The entity type." },
        { "name": "EntityName", "type": "string", "versions": "0+",
          "about": "The name of the entity." }
      ]}
    ]}
  ]
}


TODO: ConfigCommand changes.

Compatibility, Deprecation, and Migration Plan

All changes would be forward-compatible, and no migration plan is necessary. It's outside the scope of this KIP to deprecate any functionality.

Rejected Alternatives

TODO



  • No labels