You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 10 Next »

Status

Current state: Under Discussion

Discussion thread: here

JIRA: https://issues.apache.org/jira/browse/KAFKA-9494

PR: 

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

The admin client allows users to describe the configuration of brokers and topics in the cluster via describeConfigs method. The returned configuration includes name, default values, source, configuration synonym including if the information is sensitive isSensitive, or is read only isReadOnly. The consumer of this API can choose to display this information in GUI e.g. web forms in Kafka lenses, Confluent Control Center. While displaying this information, fields such as  isSensitive  and isReadOnly can be used by the client to hide sensitive information and to prevent the user from editing the field respectively. This provides better upfront experience to the user. 

This KIP proposes to include additional metadata information - dataType, documentation and validator for individual config entries. 

The motivation behind extending the schema to include dataType of the configuration entry is to enable the client to provide better validation to the user. In the absence of dataType the only way to know if the user specified value is correct is to make an `alter` call and check if there is no error, which is a suboptimal experience.  Including the dataType could open up potentially other use cases as well.

Including documentation would enable the client to provide latest information about the config and would also act as a single source of truth.

Including validator would enable the client to further improve upon the validation provided by dataType. Validator here is the serialized version of Range, ValidList, ValidString, CaseInsensitiveValidString, NonNullValidator, NonEmptyString to name a few.

In this proposal we modify the DescribeConfigsResponse schema to include the field type. 

Public Interfaces

Network protocol

Request

There is no change in the request schema except 

  1. Bump up the valid versions range
{
  "apiKey": 32,
  "type": "request",
  "name": "DescribeConfigsRequest",
  // Version 1 adds IncludeSynoyms.
  // Version 2 is the same as version 1.
  // Version 3 is the same as version 2.
  "validVersions": "0-3", <--- Updated Field
  "flexibleVersions": "none",
  "fields": [
    { "name": "Resources", "type": "[]DescribeConfigsResource", "versions": "0+",
      "about": "The resources whose configurations we want to describe.", "fields": [
      { "name": "ResourceType", "type": "int8", "versions": "0+",
        "about": "The resource type." },
      { "name": "ResourceName", "type": "string", "versions": "0+",
        "about": "The resource name." },
      { "name": "ConfigurationKeys", "type": "[]string", "versions": "0+", "nullableVersions": "0+",
        "about": "The configuration keys to list, or null to list all configuration keys." }
    ]},
    { "name": "IncludeSynoyms", "type": "bool", "versions": "1+", "default": "false", "ignorable": false,
      "about": "True if we should include all synonyms." }
  ]
}


Response

The changes to the Response schema include

  1. Bump valid versions range
  2. Add new Fields -  ConfigValueType, Documentation & Validator
{
  "apiKey": 32,
  "type": "response",
  "name": "DescribeConfigsResponse",
  "validVersions": "0-3", <--- Updated Field
  "flexibleVersions": "none",
  "fields": [
    { "name": "ThrottleTimeMs", "type": "int32", "versions": "0+",
      "about": "The duration in milliseconds for which the request was throttled due to a quota violation, or zero if the request did not violate any quota." },
    { "name": "Results", "type": "[]DescribeConfigsResult", "versions": "0+",
      "about": "The results for each resource.", "fields": [
      { "name": "ErrorCode", "type": "int16", "versions": "0+",
        "about": "The error code, or 0 if we were able to successfully describe the configurations." },
      { "name": "ErrorMessage", "type": "string", "versions": "0+", "nullableVersions": "0+",
        "about": "The error message, or null if we were able to successfully describe the configurations." },
      { "name": "ResourceType", "type": "int8", "versions": "0+",
        "about": "The resource type." },
      { "name": "ResourceName", "type": "string", "versions": "0+",
        "about": "The resource name." },
      { "name": "Configs", "type": "[]DescribeConfigsResourceResult", "versions": "0+",
        "about": "Each listed configuration.", "fields": [
        { "name": "Name", "type": "string", "versions": "0+",
          "about": "The configuration name." },
        { "name": "Value", "type": "string", "versions": "0+", "nullableVersions": "0+",
          "about": "The configuration value." },
        { "name": "ReadOnly", "type": "bool", "versions": "0+",
          "about": "True if the configuration is read-only." },
        { "name": "IsDefault", "type": "bool", "versions": "0",
          "about": "True if the configuration is not set." },
        // Note: the v0 default for this field that should be exposed to callers is
        // context-dependent. For example, if the resource is a broker, this should default to 4.
        // -1 is just a placeholder value.
        { "name": "ConfigSource", "type": "int8", "versions": "1+", "default": "-1", "ignorable": true,
          "about": "The configuration source." },
        { "name": "IsSensitive", "type": "bool", "versions": "0+",
          "about": "True if this configuration is sensitive." },
        { "name": "Synonyms", "type": "[]DescribeConfigsSynonym", "versions": "1+", "ignorable": true,
          "about": "The synonyms for this configuration key.", "fields": [
          { "name": "Name", "type": "string", "versions": "1+",
            "about": "The synonym name." },
          { "name": "Value", "type": "string", "versions": "1+", "nullableVersions": "0+",
            "about": "The synonym value." },
          { "name": "Source", "type": "int8", "versions": "1+",
            "about": "The synonym source." }
        ]},
<--- New Field
        { "name": "ConfigValueType", "type": "string", "versions": "3+", "nullableVersions": "0+",
          "about": "The configuration data type." },
        { "name": "Documentation", "type": "string", "versions": "3+", "nullableVersions": "0+",
          "about": "The configuration documentation." },
        { "name": "Validator", "type": "string", "versions": "3+", "nullableVersions": "0+",
          "about": "The valid values for the configuration." }
<---- End of New Field
      ]}
    ]}
  ]
}


AdminClient Class

No new method will be added or updated to the AdminCient class. 

Proposed Changes

Under the proposed change, AdminClient.describeConfigs response (DescribeConfigsResponse.ConfigEntry) would include following new properties

  1.  Type - This directly maps to the Type enum as defined in ConfigDef.java

 /**
 * The config types
 */
 public enum Type {
   BOOLEAN, STRING, INT, SHORT, LONG, DOUBLE, LIST, CLASS, PASSWORD
 }

      2. Documentation - field's documentation

      3. Validator - Serialized version of Range, ValidList, ValidString, CaseInsensitiveValidString, NonNullValidator, NonEmptyString, and CompositeValidator.


DescribeConfigsResponse.ConfigEntry

public class ConfigEntry {
    private final String name;
    private final String value;
    private final ConfigSource source;
    private final boolean isSensitive;
    private final boolean isReadOnly;
    private final String type; // This is the new property
    private final List<ConfigSynonym> synonyms;
	private final String documentation; // This is the new property
    private final String validator; // This is the new property
.....

Rest of the changes would be plumbing this new property in AdminManager class to be eventually consumed by AdminClient class

Sample Response

A few examples of DescribeConfigResponse 

{
  "name": "offsets.topic.num.partitions",
  "value": "50",
  "source": "DEFAULT_CONFIG",
  "type": "INT",   // New
  "synonyms": [
    {
      "name": "offsets.topic.num.partitions",
      "value": "50",
      "source": "DEFAULT_CONFIG"
    }
  ],
  "documentation": "The number of partitions for the offset commit topic (should not change after deployment)", // New
  "isReadOnly": true,
  "isSensitive": false,
  "validValues": {  // New
    "min": 1.0,
    "max": null
  }
}
{
  "name": "log.message.timestamp.type",
  "value": "CreateTime",
  "source": "DEFAULT_CONFIG",
  "type": "STRING", // New
  "synonyms": [
    {
      "name": "log.message.timestamp.type",
      "value": "CreateTime",
      "source": "DEFAULT_CONFIG"
    }
  ],
  "documentation": "Define whether the timestamp in the message is message create time or log append time. The value should be either `CreateTime` or `LogAppendTime`", // New
  "isReadOnly": false,
  "isSensitive": false,
  "validValues": {  // New
    "caseSensitive": false,
    "values": [
      "CreateTime",
      "LogAppendTime"
    ]
  }
},

Note on Recommender interface

Instead of serializing the Validator, we could leverage Recommender in org.apache.kafka.common.config.ConfigDef.ConfigKey. Few reason for not doing that

  1. Recommender is not used by any of the KafkaConfig today. 

  2. Recommender.validValues() which returns valid set of values has limited expressiveness when it comes to defining Range, nullability, case sensitiveness that comes with Validator

Compatibility, Deprecation, and Migration Plan

The proposed change is backward compatible in a sense that for same version of Kafka server and client response would include additional fields

replication.quota.window.num=11 type=INT sensitive=false synonyms={}


For server version < client version type would be null

request.timeout.ms=30000 type=null sensitive=false synonyms={}


And for server version > client version type will be ignored

Rejected Alternatives

NA

  • No labels