Table of Contents |
---|
Status
Current state: Under DiscussionAccepted
Discussion thread: here
JIRA: https://issues.apache.org/jira/browse/ Jira server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key KAFKA-9494
PR:
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
...
This KIP proposes to include 2 additional metadata information - dataType
, and documentation
and validator
for the individual config entries.
The motivation behind extending the schema to include dataType
of the configuration entry including dataType
is to enable the client to provide better validation to the user. In the absence of dataType
the only way to know if the user specified value is correct is to make an `alter` call and check if there is no error, which is a suboptimal sub-optimal experience. Including the dataType
could open up potentially other use cases as well.
Including documentation
would enable the client to provide latest information about the config and would also act as a single source of truth.
Including validator
would enable the client to further improve upon the validation provided by dataType
. Validator
here is the serialized version of Range
, ValidList
, ValidString,
CaseInsensitiveValidString
, NonNullValidator
, NonEmptyString
to name a few.
In this proposal we modify the DescribeConfigsResponse
schema to include the field type.
Public Interfaces
Network protocol
Request
There is no change in the request schema except
- Bump up the valid versions range
Public Interfaces
Network protocol
Request
The changes to the Request schema include
- Bump up the valid versions range
- New field
IncludeDocumentation
- with default value of false
Code Block |
---|
{ |
Code Block |
{ "apiKey": 32, "type": "request", "name": "DescribeConfigsRequest", // Version 1 adds IncludeSynoyms. // Version 2 is the same as version 1. // Version 3 is the same as version 2. "validVersions": "0-3", <--- Updated Field "flexibleVersions": "none", "fields": [ { "name": "Resources", "type": "[]DescribeConfigsResource", "versions": "0+", "about": "The resources whose configurations we want to describe.", "fields": [ { "name": "ResourceType", "type": "int8", "versions": "0+", "about": "The resource type." }, { "name": "ResourceName", "type": "string", "versions": "0+", "about": "The resource name." }, { "name": "ConfigurationKeys", "type": "[]string", "versions": "0+", "nullableVersions": "0+", "about": "The configuration keys to list, or null to list all configuration keys." } ]}, { "name": "IncludeSynoyms", "type": "bool", "versions": "1+", "default": "false", "ignorable": false, "about": "True if we should include all synonyms." } { ] } |
Response
The changes to the Response schema include
- Bump valid versions range
- Add new Fields -
ConfigValueType, Documentation & Validator
Code Block |
---|
{ "apiKey": 32, "type": "response", "name": "DescribeConfigsResponse", "validVersions": "0-3", "name": "IncludeDocumentation", "type": "bool", "versions": "1+", "default": "false", "ignorable": false, <--- UpdatedNew Field "flexibleVersions": "none", "fieldsabout": [ "True if we should include configuration documentation." } ] } |
Response
The changes to the Response schema include
- Bump valid versions range
- Add new Fields -
ConfigValueType and Documentation
Code Block |
---|
{ "apiKey": 32, "type": "response", "name": "DescribeConfigsResponse", "validVersions": "0-3", <--- Updated Field "flexibleVersions": "none", "fields": [{ "name": "ThrottleTimeMs", "type": "int32", "versions": "0+", "about": "The duration in milliseconds for which the request was throttled due to a quota violation, or zero if the request did not violate any quota." }, { "name": "ResultsThrottleTimeMs", "type": "[]DescribeConfigsResultint32", "versions": "0+", "about": "The resultsduration for each resource.", "fields": [ in milliseconds for which the request was throttled due to a quota violation, or zero if the request did not violate any quota." }, { "name": "ErrorCodeResults", "type": "int16[]DescribeConfigsResult", "versions": "0+", "about": "The errorresults code,for or 0 if we were able to successfully describe the configurations." },each resource.", "fields": [ { "name": "ErrorMessageErrorCode", "type": "stringint16", "versions": "0+", "nullableVersions": "0+", "about": "The error messagecode, or null0 if we were able to successfully describe the configurations." }, { "name": "ResourceTypeErrorMessage", "type": "int8string", "versions": "0+", "nullableVersions": "0+", "about": "The resource type error message, or null if we were able to successfully describe the configurations." }, { "name": "ResourceNameResourceType", "type": "stringint8", "versions": "0+", "about": "The resource nametype." }, { "name": "ConfigsResourceName", "type": "[]DescribeConfigsResourceResultstring", "versions": "0+", "about": "EachThe listedresource configurationname." }, " { "name": "Configs", "type": "[]DescribeConfigsResourceResult", "versions": "0+", "about": "Each listed configuration.", "fields": [ { "name": "Name", "type": "string", "versions": "0+", "about": "The configuration name." }, { "name": "Value", "type": "string", "versions": "0+", "nullableVersions": "0+", "about": "The configuration value." }, { "name": "ReadOnly", "type": "bool", "versions": "0+", "about": "True if the configuration is read-only." }, { "name": "IsDefault", "type": "bool", "versions": "0", "about": "True if the configuration is not set." }, // Note: the v0 default for this field that should be exposed to callers is // context-dependent. For example, if the resource is a broker, this should default to 4. // -1 is just a placeholder value. { "name": "ConfigSource", "type": "int8", "versions": "1+", "default": "-1", "ignorable": true, "about": "The configuration source." }, { "name": "IsSensitive", "type": "bool", "versions": "0+", "about": "True if this configuration is sensitive." }, { "name": "Synonyms", "type": "[]DescribeConfigsSynonym", "versions": "1+", "ignorable": true, "about": "The synonyms for this configuration key.", "fields": [ { "name": "Name", "type": "string", "versions": "1+", "about": "The synonym name." }, { "name": "Value", "type": "string", "versions": "1+", "nullableVersions": "0+", "about": "The synonym value." }, { "name": "Source", "type": "int8", "versions": "1+", "about": "The synonym source." } ]}, <--- New Field { "name": "ConfigValueType", "type": "stringint8", "versions": "3+", "nullableVersionsdefault": "0+", <--- New Field "about": "The configuration data type." }, { "name": "Documentation", "type": "string", "versions": "3+", "nullableVersions": "0+", <--- New Field "about": "The configuration documentation." }, ]} { "name": "Validator", "type": "string", "versions": "3+", "nullableVersions": "0+", "about": "The valid values for the configuration." } <---- End of New Field ]} ]} ] } ]} ] } |
AdminClient Class
No new method will be added or updated to the AdminCient
class.
Proposed Changes
Under the proposed change, AdminClient.describeConfigs
response (DescribeConfigsResponse.ConfigEntry
) would include following new properties
...
AbstractConfig Class
As part of this KIP, a new method will be added to AbstractConfig to return the documentation of the given Config
Code Block |
---|
public String documentationOf(String key) |
Code Block |
/** * The config types */ public enum Type { BOOLEAN, STRING, INT, SHORT, LONG, DOUBLE, LIST, CLASS, PASSWORD } |
2. Documentation
- field's documentation
3. Validator
- Serialized version of Range
, ValidList
, ValidString,
CaseInsensitiveValidString
, NonNullValidator
, NonEmptyString, and CompositeValidator
.
DescribeConfigsResponse.ConfigEntry
Code Block |
---|
public class ConfigEntry {
private final String name;
private final String value;
private final ConfigSource source;
private final boolean isSensitive;
private final boolean isReadOnly;
private final String type; // This is the new property
private final List<ConfigSynonym> synonyms;
private final String documentation; // This is the new property
private final String validator; // This is the new property
..... |
Rest of the changes would be plumbing this new property in AdminManager
class to be eventually consumed by AdminClient
class
Sample Response
A few examples of DescribeConfigResponse
...
language | js |
---|
...
ConfigDef.ConfigKey configKey = definition.configKeys().get(key);
if (configKey == null)
return null;
return configKey.documentation;
} |
Proposed Changes
Under the proposed change, response for AdminClient.describeConfigs
would include following new properties, which would be part of DescribeConfigsResponse.ConfigEntry
class.
Type: ConfigType -
This is an Enum of all the supported data type.Code Block /** * Data type of configuration entry. */ public enum ConfigType { UNKNOWN, BOOLEAN, STRING, INT, SHORT, LONG, DOUBLE, LIST, CLASS, PASSWORD }
ConfigType.Unknown
is for ensuring backward compatibility (explained is Compatibility section below)
The values in above Enum is derived from existingType
Enum defined in ConfigDef.java belowCode Block /** * The config types */ public enum Type { BOOLEAN, STRING, INT, SHORT, LONG, DOUBLE, LIST, CLASS, PASSWORD }
2. Documentation: String
- field's documentation
DescribeConfigsResponse.ConfigEntry
with the above new fields
Code Block |
---|
public class ConfigEntry {
private final String name;
private final String value;
private final ConfigSource source;
private final boolean isSensitive;
private final boolean isReadOnly;
private final List<ConfigSynonym> synonyms;
.....
private final ConfigType type; // This is the new property
private final String documentation; // This is the new property
..... |
Rest of the changes would be plumbing this new property in AdminManager
class to be eventually consumed by AdminClient
class
Sample Response
Following example show how the response would look like in JSON format.
Note: Kafka uses binary protocol over TCP. ConfigType Enum is serialized as byte instead of string.
Code Block | ||
---|---|---|
| ||
{ "name": "logoffsets.messagetopic.timestampnum.typepartitions", "value": "CreateTime50", "source": "DEFAULT_CONFIG", "type": "STRINGINT", // <---- New Field "synonyms": [ { "name": "logoffsets.messagetopic.timestampnum.typepartitions", "value": "CreateTime50", "source": "DEFAULT_CONFIG" } ], "documentation": "DefineThe whethernumber theof timestamppartitions infor the messageoffset iscommit messagetopic create(should timenot orchange log append time. The value should be either `CreateTime` or `LogAppendTime`", // Newafter deployment)", // <---- New Field "isReadOnly": falsetrue, "isSensitive": false, "validValues": { // New "caseSensitive": false, "values": [ } |
Default Behavior
By default, the `documentation` field will not be included in the response to describeConfigs
. This is to avoid the response from boating up as the value for documentation
field can be quite large.
Below example shows how client can include the new fields in the response
Code Block | ||
---|---|---|
| ||
KafkaAdminClient.describeConfigs( resources, new DescribeConfigsOptions() "CreateTime", "LogAppendTime" .includeSynonyms(true) ] } }, |
Note on Recommender interface
Instead of serializing the Validator
, we could leverage Recommender in org.apache.kafka.common.config.ConfigDef.ConfigKey.
Few reason for not doing that
- Recommender is not used by any of the
KafkaConfig
today. Recommender.validValues()
which returns valid set of values has limited expressiveness when it comes to defining Range, nullability, case sensitiveness that comes withValidator
Compatibility, Deprecation, and Migration Plan
The proposed change is backward compatible in a sense that for same version of Kafka server and client response would include additional fields
Code Block |
---|
replication.quota.window.num=11 type=INT sensitive=false synonyms={} |
...
Code Block |
---|
request.timeout.ms=30000 type=null sensitive=false synonyms={} |
...
.includeDocumentation(true)
) |
Compatibility, Deprecation, and Migration Plan
For backward compatibility we rely on existing schema versioning technique. Under this, client sends its version information in the request header. This in turn is used on the server to select the schema for that version.
Server version | Client version | Expected Behavior | |
---|---|---|---|
1 | X | X | Both Type and Documentation is visible to the client. |
2 | X | X-1 |
Not serialized as part of the response. |
3 | X-1 | X | IllegalArgumentException: Invalid version for API key... |
4 | X+1 | X | Both If version X+1 has new value in enum |
Rejected Alternatives
NA