THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
This KIP proposes to include additional metadata information - dataType, and documentation and validator for individual config entries.
...
Including documentation would enable the client to provide latest information about the config and would also act as a single source of truth.
Including validator would enable the client to further improve upon the validation provided by dataType. Validator here is the serialized version of Range, ValidList, ValidString, CaseInsensitiveValidString, NonNullValidator, NonEmptyString to name a few.
In this proposal we modify the DescribeConfigsResponse schema to include the field type.
Public Interfaces
Network protocol
Request
There is no change in the request schema except
Public Interfaces
Network protocol
Request
There is no change in the request schema except
- Bump up Bump up the valid versions range
| Code Block |
|---|
{
"apiKey": 32,
"type": "request",
"name": "DescribeConfigsRequest",
// Version 1 adds IncludeSynoyms.
// Version 2 is the same as version 1.
// Version 3 is the same as version 2.
"validVersions": "0-3", <--- Updated Field
"flexibleVersions": "none",
"fields": [
{ "name": "Resources", "type": "[]DescribeConfigsResource", "versions": "0+",
"about": "The resources whose configurations we want to describe.", "fields": [
{ "name": "ResourceType", "type": "int8", "versions": "0+",
"about": "The resource type." },
{ "name": "ResourceName", "type": "string", "versions": "0+",
"about": "The resource name." },
{ "name": "ConfigurationKeys", "type": "[]string", "versions": "0+", "nullableVersions": "0+",
"about": "The configuration keys to list, or null to list all configuration keys." }
]},
{ "name": "IncludeSynoyms", "type": "bool", "versions": "1+", "default": "false", "ignorable": false,
"about": "True if we should include all synonyms." }
]
} |
...
- Bump valid versions range
- Add new Fields -
ConfigValueType, Documentation & Validatorand Documentation
| Code Block |
|---|
{
"apiKey": 32,
"type": "response",
"name": "DescribeConfigsResponse",
"validVersions": "0-3", <--- Updated Field
"flexibleVersions": "none",
"fields": [
{ "name": "ThrottleTimeMs", "type": "int32", "versions": "0+",
"about": "The duration in milliseconds for which the request was throttled due to a quota violation, or zero if the request did not violate any quota." },
{ "name": "Results", "type": "[]DescribeConfigsResult", "versions": "0+",
"about": "The results for each resource.", "fields": [
{ "name": "ErrorCode", "type": "int16", "versions": "0+",
"about": "The error code, or 0 if we were able to successfully describe the configurations." },
{ "name": "ErrorMessage", "type": "string", "versions": "0+", "nullableVersions": "0+",
"about": "The error message, or null if we were able to successfully describe the configurations." },
{ "name": "ResourceType", "type": "int8", "versions": "0+",
"about": "The resource type." },
{ "name": "ResourceName", "type": "string", "versions": "0+",
"about": "The resource name." },
{ "name": "Configs", "type": "[]DescribeConfigsResourceResult", "versions": "0+",
"about": "Each listed configuration.", "fields": [
{ "name": "Name", "type": "string", "versions": "0+",
"about": "The configuration name." },
{ "name": "Value", "type": "string", "versions": "0+", "nullableVersions": "0+",
"about": "The configuration value." },
{ "name": "ReadOnly", "type": "bool", "versions": "0+",
"about": "True if the configuration is read-only." },
{ "name": "IsDefault", "type": "bool", "versions": "0",
"about": "True if the configuration is not set." },
// Note: the v0 default for this field that should be exposed to callers is
// context-dependent. For example, if the resource is a broker, this should default to 4.
// -1 is just a placeholder value.
{ "name": "ConfigSource", "type": "int8", "versions": "1+", "default": "-1", "ignorable": true,
"about": "The configuration source." },
{ "name": "IsSensitive", "type": "bool", "versions": "0+",
"about": "True if this configuration is sensitive." },
{ "name": "Synonyms", "type": "[]DescribeConfigsSynonym", "versions": "1+", "ignorable": true,
"about": "The synonyms for this configuration key.", "fields": [
{ "name": "Name", "type": "string", "versions": "1+",
"about": "The synonym name." },
{ "name": "Value", "type": "string", "versions": "1+", "nullableVersions": "0+",
"about": "The synonym value." },
{ "name": "Source", "type": "int8", "versions": "1+",
"about": "The synonym source." }
]},
<--- Start Newof new FieldFields
{ "name": "ConfigValueType", "type": "string", "versions": "3+", "nullableVersions": "0+",
"about": "The configuration data type." },
Type can be one of the following values - BOOLEAN, STRING, INT, SHORT, LONG, DOUBLE, LIST, CLASS, PASSWORD"},
{ "name": "Documentation", "type": "string", "versions": "3+", "nullableVersions": "0+",
"about": "The configuration documentation." },
<---- End of new Field
{ "name": "Validator", "type": "string", "versions": "3+", "nullableVersions": "0+",
"about": "The valid values for the configuration." }
<---- End of New Field
]}
]}
]
}
|
AdminClient Class
No new method will be added or updated to the AdminCient class.
Proposed Changes
Under the proposed change, AdminClient.describeConfigs response (DescribeConfigsResponse.ConfigEntry) would include following new properties
...
]}
]}
]
}
|
AdminClient Class
No new method will be added or updated to the AdminCient class.
Proposed Changes
Under the proposed change, AdminClient.describeConfigs response (DescribeConfigsResponse.ConfigEntry) would include following new properties
-
Type -This directly maps to theTypeenum as defined in ConfigDef.java
| Code Block |
|---|
/**
* The config types
*/
public enum Type {
BOOLEAN, STRING, INT, SHORT, LONG, DOUBLE, LIST, CLASS, PASSWORD
} |
2. Documentation - field's documentation
DescribeConfigsResponse.ConfigEntry
| Code Block |
|---|
public class ConfigEntry {
private final String name;
private final String value;
private final ConfigSource source;
private final boolean isSensitive |
| Code Block |
|---|
/**
* The config types
*/
public enum Type {
BOOLEAN, STRING, INT, SHORT, LONG, DOUBLE, LIST, CLASS, PASSWORD
} |
2. Documentation - field's documentation
3. Validator - Serialized version of Range, ValidList, ValidString, CaseInsensitiveValidString, NonNullValidator, NonEmptyString, and CompositeValidator.
DescribeConfigsResponse.ConfigEntry
| Code Block |
|---|
public class ConfigEntry { private final String name; private final String value; private final ConfigSource source; private final boolean isSensitiveisReadOnly; private final booleanList<ConfigSynonym> isReadOnlysynonyms; private final String type; // This is the new property private final List<ConfigSynonym> synonyms; private final String documentation; // This is the new property private final String validator; // This is the new property .......... |
Rest of the changes would be plumbing this new property in AdminManager class to be eventually consumed by AdminClient class
Sample Response
Kafka uses binary protocol over TCP. Following example, however show how the response would look like in JSON format.
A few examples of DescribeConfigResponse
| Code Block | ||
|---|---|---|
| ||
{
"name": "offsets.topic.num.partitions",
"value": "50",
"source": "DEFAULT_CONFIG",
"type": "INT", // New Fild
"synonyms": [
{
"name": "offsets.topic.num.partitions",
"value": "50",
"source": "DEFAULT_CONFIG"
}
],
"documentation": "The number of partitions for the offset commit topic (should not change after deployment)", // New Field
"isReadOnly": true,
"isSensitive": false,
"validValues": { // New
"min": 1.0,
"max": null
}
} |
| Code Block | ||
|---|---|---|
| ||
{
"name": "log.message.timestamp.type",
"value": "CreateTime",
"source": "DEFAULT_CONFIG",
"type": "STRING", // New
"synonyms": [
{
"name": "log.message.timestamp.type",
"value": "CreateTime",
"source": "DEFAULT_CONFIG"
}
],
"documentation": "Define whether the timestamp in the message is message create time or log append time. The value should be either `CreateTime` or `LogAppendTime`", // New
"isReadOnly": false,
"isSensitive": false,
"validValues": { // New
"caseSensitive": false,
"values": [
"CreateTime",
"LogAppendTime"
]
}
}, |
Note on Recommender interface
Instead of serializing the Validator, we could leverage Recommender in org.apache.kafka.common.config.ConfigDef.ConfigKey. Few reason for not doing that
...
,
} |
Compatibility, Deprecation, and Migration Plan
The proposed change is backward compatible in a sense that for same version of Kafka server and client response would include additional fields
For example, if one runs kafka-config.sh
bin/kafka-configs.sh --describe --entity-type brokers --bootstrap-server localhost:9092 --entity-name 0
the output would be asThe proposed change is backward compatible in a sense that for same version of Kafka server and client response would include additional fields
| Code Block |
|---|
replication.quota.window.num=11 type=INT sensitive=false synonyms={} |
...
And for server version > client version type will be ignored
| Code Block |
|---|
request.timeout.ms=30000 sensitive=false synonyms={} |
Rejected Alternatives
NA