Versions Compared

Old Version 4

changes.mady.by.user Unknown User (showuon)

Saved on

compared with

New Version 5

changes.mady.by.user Unknown User (showuon)

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

The table lists current configs that fit the above 2 criteria:

existing confidential changable confignewly added internal configsdefault valueupdate mode
sasl.jaas.configsasl.jaas.config.timestamp-1per-broker
ssl.keystore.keyssl.keystore.key.timestamp-1per-broker
ssl.keystore.passwordssl.keystore.password.timestamp-1per-broker
ssl.keystore.certificate.chainssl.keystore.certificate.chain.timestamp-1per-broker
ssl.truststore.certificatesssl.truststore.certificates.timestamp-1per-broker
ssl.truststore.passwordssl.truststore.password.timestamp-1per-broker
ssl.key.passwordssl.key.password.timestamp-1per-broker

For these configs, there will be an "timestamp" config added. Ex: "ssl.keystore.key" will add "ssl.keystore.key.timestamp" as an internal config with default value -1. It has the same update mode (per-broker). The internal config means it is not publicly visible. And users cannot alter values via admin API or broker properties. Besides, there will be a new ConfigDef.Type added: METADATA, to allow brokers restrict users update these configs.

...

Code Block
    /**
     * The type for a configuration value
     */
    public enum Type {
        /**
         * Used for boolean values. Values can be provided as a Boolean object or as a String with values
         * <code>true</code> or <code>false</code> (this is not case-sensitive), otherwise a {@link ConfigException} is
         * thrown.
         */
        BOOLEAN,
        ...

        /**
         * Used for string values containing sensitive data such as a password or key. The values of configurations with
         * of this type are not included in logs and instead replaced with "[hidden]". Values must be provided as a
         * String object, otherwise a {@link ConfigException} is thrown.
         */
        PASSWORD,
    
        // new
ly added below
        /**
         * Used for string values containing the metadata of the according sensitive field such as a password or key. 
         * Values must be provided as a String object, otherwise a {@link ConfigException} is thrown.
         */
        METADATA;

DescribeConfigs

Request:

Bump the version to 5 because of DescribeConfigsResponse change.

Response:

Added "LastUpdatedTimestamp" field


Restriction for METADATA type configs:

(1) alter

These METADATA type configs cannot be updated via admin API, otherwise, exception will be thrown.

Code Block
> bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type brokers --entity-name 2 --alter --add-config listener.name.ssl.ssl.keystore.key.timestamp=123

Error while executing config command with args '--bootstrap-server localhost:9092 --entity-type brokers --entity-name 2 --alter --add-config listener.name.ssl.ssl.keystore.key.timestamp=123'
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidRequestException: Cannot update metadata type config:HashMap(listener.name.ssl.ssl.keystore.key.timestamp -> 123)


Metadata type config update:

The metadata type config can only be updated when the mapping config getting updated. For example:

Code Block
> bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type brokers --entity-name 2 --alter --add-config listener.name.ssl.ssl.key.password=000000

Completed updating config for broker 2.


Since the config: listener.name.ssl.ssl.key.password  is a confidential chanagable config, in the controller we'll also add one more entry to the metadata log to update listener.name.ssl.ssl.key.password.timestamp metadata config, and set the value as current timestamp in milliseconds. When checking the controller log, you'll see these lines:


Code Block
[2024-11-15 14:26:10,963] INFO [QuorumController id=1] Replayed ConfigRecord for ConfigResource(type=BROKER, name='2') which set configuration listener.name.ssl.ssl.key.password to null (org.apache.kafka.controller.ConfigurationControlManager)

// added by controller 
[2024-11-15 14:26:10,964] INFO [QuorumController id=1] Replayed ConfigRecord for ConfigResource(type=BROKER, name='2') which set configuration listener.name.ssl.ssl.key.password.timestamp to 1731651970963 (org.apache.kafka.controller.ConfigurationControlManager)


(2) describe

Because the metadata configs are internal configs, it cannot be shown as an config entry when describing configs. Instead, the metadata value will be injected into the mapping confidential config as "lastUpdatedTimestamp" field. When describing the config, we'll see it:


Code Block
> bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type brokers --entity-name 2  --describe --all

// non-sensitive field will not show "lastUpdatedTimestamp" field
advertised.listeners=PLAINTEXT://localhost:9092 sensitive=false synonyms={STATIC_BROKER_CONFIG:advertised.listeners=PLAINTEXT://localhost:9092}
...
// sensitive field will show "lastUpdatedTimestamp" field. If it is not changed, it'll be -1.
listener.name.sslnochange.ssl.key.password=null sensitive=true synonyms={STATIC_BROKER_CONFIG:listener.name.plaintext.ssl.key.password=null, STATIC_BROKER_CONFIG:ssl.key.password=null} lastUpdatedTimestamp=-1
// If it is changed, it'll return the last updated timestamp
listener.name.ssl.ssl.key.password=null sensitive=true synonyms={STATIC_BROKER_CONFIG:listener.name.ssl.ssl.key.password=null, STATIC_BROKER_CONFIG:ssl.key.password=null} lastUpdatedTimestamp=1731651970963

Thus, in DescribeConfigsResponse, we need to add one "LastUpdatedTimestamp" field to it.


DescribeConfigs

Request:

Bump the version to 5 because of DescribeConfigsResponse change.

Response:

Bump version to 5 to add "LastUpdatedTimestamp" field

Code Block
{
  "apiKey": 32,
  "type": "response",
  "name": "DescribeConfigsResponse",
  // Version 1 adds ConfigSource and the synonyms.
  // Starting in version 2, on quota violation, brokers send out responses before throttling.
  // Version 4 enables flexible versions.
  // Version 5 adds "LastUpdateTimestamp" field
  "validVersions": "0-5",
  "flexibleVersions": "5+",
  "fields": [
    { "name"
Code Block
{
  "apiKey": 32,
  "type": "response",
  "name": "DescribeConfigsResponse",
  // Version 1 adds ConfigSource and the synonyms.
  // Starting in version 2, on quota violation, brokers send out responses before throttling.
  // Version 4 enables flexible versions.
  // Version 5 adds "LastUpdateTimestamp" field
  "validVersions": "0-5",
  "flexibleVersions": "5+",
  "fields": [
    { "name": "ThrottleTimeMs", "type": "int32", "versions": "0+",
      "about": "The duration in milliseconds for which the request was throttled due to a quota violation, or zero if the request did not violate any quota." },
    { "name": "Results", "type": "[]DescribeConfigsResult", "versions": "0+",
      "about": "The results for each resource.", "fields": [
      { "name": "ErrorCode", "type": "int16", "versions": "0+",
        "about": "The error code, or 0 if we were able to successfully describe the configurations." },
      { "name": "ErrorMessage", "type": "string", "versions": "0+", "nullableVersions": "0+",
        "about": "The error message, or null if we were able to successfully describe the configurations." },
      { "name": "ResourceType", "type": "int8", "versions": "0+",
        "about": "The resource type." },
      { "name": "ResourceName", "type": "string", "versions": "0+",
        "about": "The resource name." },
      { "name": "Configs", "type": "[]DescribeConfigsResourceResult", "versions": "0+",
        "about": "Each listed configuration.", "fields": [
        { "name": "Name", "type": "string", "versions": "0+",
          "about": "The configuration name." },
        { "name": "Value", "type": "string", "versions": "0+", "nullableVersions": "0+",
          "about": "The configuration value." },
        { "name": "ReadOnly", "type": "bool", "versions": "0+",
          "about": "True if the configuration is read-only." },
        { "name": "IsDefault", "type": "bool", "versions": "0",
          "about": "True if the configuration is not set." },
        // Note: the v0 default for this field that should be exposed to callers is
        // context-dependent. For example, if the resource is a broker, this should default to 4.
        // -1 is just a placeholder value.
        { "name": "ConfigSource", "type": "int8", "versions": "1+", "default": "-1", "ignorable": true,
 broker, this should default to 4.
        // -1 is "about": "The configuration source." },just a placeholder value.
        { "name": "ConfigSource", "nametype": "IsSensitiveint8", "typeversions": "bool1+", "versionsdefault": "-1", "0+ignorable": true,
          "about": "True if thisThe configuration is sensitivesource." },
        { "name": "SynonymsIsSensitive", "type": "[]DescribeConfigsSynonymbool", "versions": "10+", "ignorable": true,
          "about": "TheTrue synonyms forif this configuration is keysensitive.", "fields": [},
          { "name": "NameSynonyms", "type": "string[]DescribeConfigsSynonym", "versions": "1+",
 "ignorable": true,
          "about": "The synonym name synonyms for this configuration key." },, "fields": [
          { "name": "ValueName", "type": "string", "versions": "1+", "nullableVersions": "0+",
            "about": "The synonym valuename." },
          { "name": "SourceValue", "type": "string", "versions": "int81+", "versionsnullableVersions": "10+",
            "about": "The synonym sourcevalue." },
        ]},
        { "name": "ConfigTypeSource", "type": "int8", "versions": "31+",
 "default": "0", "ignorable": true,
          "about": "The configuration data type. Type can be one of the following values - BOOLEAN, STRING, INT, SHORT, LONG, DOUBLE, LIST, CLASS, PASSWORD" The synonym source." }
        ]},
        { "name": "DocumentationConfigType", "type": "stringint8", "versions": "3+", "nullableVersionsdefault": "0+", "ignorable": true,
          "about": "The configuration data documentation." },

        // newly added fieldtype. Type can be one of the following values - BOOLEAN, STRING, INT, SHORT, LONG, DOUBLE, LIST, CLASS, PASSWORD" },
        { "name": "LastUpdateTimestampDocumentation", "type": "int64string", "versions": "3+", "5nullableVersions": "0+", "ignorable": true,
          "about": "The last updated timestamp for this config." }
      ]}
    ]}
  ]
}

Restriction:

These METADATA type configs cannot be updated via admin API, otherwise, exception will be thrown.

Code Block
> bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type brokers --entity-name 2 --alter --add-config listener.name.ssl.ssl.keystore.key.timestamp=123

Error while executing config command with args '--bootstrap-server localhost:9092 --entity-type brokers --entity-name 2 --alter --add-config listener.name.ssl.ssl.keystore.key.timestamp=123'
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidRequestException: Cannot update metadata type config:HashMap(listener.name.ssl.ssl.keystore.key.timestamp -> 123)

Metadata type config update:

The metadata type config will only be updated when the related config getting updated. For example:

Code Block
> bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type brokers --entity-name 2 --alter --add-config listener.name.ssl.ssl.key.password=000000

Completed updating config for broker 2.

Since the config: listener.name.ssl.ssl.key.password  is a confidential chanagable config, in the controller, we'll also add one more entry to update listener.name.ssl.ssl.key.password.timestamp metadata config, and set the value as current timestamp in milliseconds.

Compatibility, Deprecation, and Migration Plan

...

 configuration documentation." },

        // newly added field
        { "name": "LastUpdateTimestamp", "type": "int64", "versions": "5+", "ignorable": true,
          "about": "The last updated timestamp for this config." }
      ]}
    ]}
  ]
}


Compatibility, Deprecation, and Migration Plan

...


Test Plan

Describe in few sentences how the KIP will be tested. We are mostly interested in system tests (since unit-tests are specific to implementation details). How will we know that the implementation works as expected? How will we know nothing broke?

...