Status
Current state: Draft
Discussion thread: TODO
JIRA: TODO
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation
Quota management via Admin Client has gone through a couple drafts of proposals (KIP-248, KIP-422). While improvements have been made to the Admin interface for configuration handling, fitting quotas into the API would be restrictive because they don't fit the natural key-value pairing. Therefore, it'd be beneficial to have a quota-native API for administrating quotas, which would offer an intuitive and less error-prone interface, convey additional useful information beyond what the configuration APIs could provide, and provide for future extensibility as quotas types are added or evolved.
Background
Quotas are defined in terms of a user and client ID, where the user acts as a specific opaque principal, and the client ID as a more generic group identifier.
When setting quotas, an administrator has flexibility in how it specifies the user and client ID for which the quota applies to, where the user and client ID may be named, indicated as the default, or omitted entirely. This creates a hierarchy structure for which quotas apply, where the most specific, defined quota will be matched to a request's user and client ID.
As represented by the current ZK node structure, the order in which quotas are matched are as follows. Note <user> is a specified user name, <client-id> is a specified client ID, and <default> is a special default user/client ID represented as the literal string "<default>".
/config/users/<user>/clients/<client-id>
/config/users/<user>/clients/<default>
/config/users/<user>
/config/users/<default>/clients/<client-id>
/config/users/<default>/clients/<default>
/config/users/<default>
/config/clients/<client-id>
/config/clients/<default>
As such, reasoning around quotas can be complex, as it's not immediately obvious which quotas may apply to a given user and/or client ID. Providing descriptive information is a goal of this KIP.
Public Interfaces
/** * Represents the default name for an entity, i.e. the entity that's matched * when an exact match isn't found. */ public final static String QUOTA_ENTITY_DEFAULT = // implementation defined /** * Describes an entity type. */ public enum QuotaEntity { USER, // The user principal. CLIENT_ID, // The client ID. // Note others may be added in the future. } public class QuotaFilter { public enum Rule { EXACT, // exact name match NOT, // matches all non-matching names PREFIX, // matches all names with the given prefix // Note others may be added in the future. } /** * A filtering rule to be applied. * * @param entity the entity the rule applies to * @param rule the rule to apply * @param match the non-null string that's applied by the rule */ public QuotaFilter(QuotaEntity entity, Rule rule, String match); public QuotaEntity entity(); public Rule rule(); public String match(); } public class DescribeQuotasOptions { // Default. } public class DescribeQuotasResult { /** * @return the collection of entities that matched the filter */ KafkaFuture<Collection<Map<QuotaEntity, String>>> entities(); } public interface Admin extends AutoCloseable { /** * Describes all entities matching all provided filters (logical AND) that have at least one * quota value defined. */ DescribeQuotasResult describeQuotas(Collection<QuotaFilter> filters, DescribeQuotasOptions options); }
/** * The quota types. */ public enum QuotaType { CONSUMER_BYTE_RATE, PRODUCER_BYTE_RATE, REQUEST_PERCENTAGE, // Note others may be added in the future. } public class DescribeEffectiveQuotasOptions { // Whether to omit any inherited values in the result. If true, then undefined values for // the entity's config will be excluded from the results. // // Default: false. DescribeQuotasOptions setOmitInheritedValues(boolean omitInheritedValues); // Whether to include the list of overridden values for every quota type in the result. // // Default: false. DescribeQuotasOptions setOmitOverriddenValues(boolean omitOverriddenValues); } public class DescribeEffectiveQuotasResult { /** * Information about a specific quota configuration value. */ public class Value { /** * @param source the entity source for the value * @param value the non-null value */ public Value(Map<QuotaEntity, String> source, Double value); public Map<QuotaEntity, String> source(); public Double value(); } /** * Information about the value for a quota type. */ public class Entry { /** * @param value the active quota value * @param overriddenValues all values that are overridden due to being lower in specificity */ public Entry(Value value, List<Value> overriddenValues); public Value value(); public List<Value> overriddenValues(); } /** * Maps a quota type to its configuration entry. * * Note that if `options.omitInheritedValues` is true, then this config may not map every * quota type to an entry. If a key is not contained in the map, then that quota type's value * is not specified. */ public KafkaFuture<Map<QuotaType, Entry>> config(); } public interface Admin extends AutoCloseable { /** * Describes the effective quotas for the provided entities. */ DescribeEffectiveQuotasResult describeEffectiveQuotas(Collection<QuotaEntity> quotaEntities, DescribeEffectiveQuotasOptions options); }
public class AlterQuotasOp { /** * @param type the quota type to alter * @param the new value for the quota, otherwise if null, then the existing value is cleared */ public QuotaAlteration(QuotaType type, Double value); public QuotaType type(); public Double value(); } public class AlterQuotasOptions { // Default. } public class AlterQuotasResult { /** * @return map of a quota entity update to its future */ public Map<QuotaEntity, KafkaFuture<Void>> futures(); } public interface Admin extends AutoCloseable { /** * Alters the quotas as specified for the provided quota entities. * * @param alterations the alterations to perform * @return the result of the alterations */ AlterQuotasResult alterQuotas(Map<QuotaEntity, AlterQuotasOp> alterations); }
Proposed Changes
Compatibility, Deprecation, and Migration Plan
All changes would be forward-compatible, and no migration plan is necessary. It's outside the scope of this KIP to deprecate any functionality.