This page is meant as a template for writing a KIP. To create a KIP choose Tools->Copy on this page and modify with your content and replace the heading with the next KIP number and a description of your issue. Replace anything in italics with your own description.

Status

Current state: "Under Discussion"

Discussion thread: here

JIRA: KAFKA-18005

Motivation

Currently, when describing config for a resource, we'll get `null` if the config is a sensitive config, ex: "ssl.keystore.certificate.chain", "ssl.keystore.password". And when describing configs with them it'll always return something like this:

> bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type brokers --entity-name 2 --describe
Dynamic configs for broker 2 are:
  listener.name.myssl.ssl.keystore.key=null sensitive=true synonyms={DYNAMIC_BROKER_CONFIG:listener.name.myssl.
=null} 

Note: The value of `listener.name.myssl.ssl.keystore.key` should not be null , but returning null for security consideration.

It makes sense to not expose sensitive data to the users, but returning null will let the users/operators hard to identify if the config is up-to-date.

Take the example in KIP-412:

Consider the dynamic configurations ssl.keystore.location, ssl.keystore.password and ssl.keystore.type are stored in vault and their values are resolved using a VaultConfigProvider. 

1. The user will update the ssl.keystore.location, ssl.keystore.password, ssl.keystore.type in the vault.
2. After the updates are complete he will send a adminClient request to the broker to notify that configs are updated.
3. Once the Broker receives a alteredConfig request it will invoke the get function in VaultConfigProvider.
4. The VaultConfigProvider will fetch the actual values for ssl.keystore.location, ssl.keystore.password, ssl.keystore.type from the vault.
5. The broker will validate these configs and apply the changes.

In the step (2), the operator needs a way to know if these configs are up-to-date. With current design, the operator will never know it, and blindly run the alter config multiple times, or worse, the operator thought it is already updated and skipped this update, and cause the broker connection failure.

It would be great if the broker can return some metadata of these sensitive configs, like "last modified timestamp", to allow the operators have a way to know if this config is up-to-date.

Public Interfaces

There will be an additional "internal config" added for each dynamic confidential configurations. The definition of dynamic confidential configurations is:

1. This config is PASSWORD type.
2. This config is a dynamically changable config.

For these configs, there will be an "timestamp" config added. Ex: "ssl.keystore.key" will add "ssl.keystore.key.timestamp" as an internal config. It has the same update mode (ex: per-broker, cluster-wide). The internal config means it is not publicly visible. And users cannot set/alter values via admin API or broker properties. Besides, there will be a new ConfigDef.Type added: METADATA, to allow brokers restrict users update these configs.

Proposed Changes

The METADATA configDef type will be added:

    /**
     * The type for a configuration value
     */
    public enum Type {
        /**
         * Used for boolean values. Values can be provided as a Boolean object or as a String with values
         * <code>true</code> or <code>false</code> (this is not case-sensitive), otherwise a {@link ConfigException} is
         * thrown.
         */
        BOOLEAN,
        ...
        /**
         * Used for string values containing sensitive data such as a password or key. The values of configurations with
         * of this type are not included in logs and instead replaced with "[hidden]". Values must be provided as a
         * String object, otherwise a {@link ConfigException} is thrown.
         */
        PASSWORD,
    
        // new added below
        /**
         * Used for string values containing the metadata of the according sensitive field such as a password or key. 
         * Values must be provided as a String object, otherwise a {@link ConfigException} is thrown.
         */
        METADATA;

By default, the timestamp metadata config value is "-1". When it is set,

Compatibility, Deprecation, and Migration Plan

• What impact (if any) will there be on existing users?
• If we are changing behavior how will we phase out the older behavior?
• If we need special migration tools, describe them here.
• When will we remove the existing behavior?

Test Plan

Describe in few sentences how the KIP will be tested. We are mostly interested in system tests (since unit-tests are specific to implementation details). How will we know that the implementation works as expected? How will we know nothing broke?

Rejected Alternatives

If there are alternative ways of accomplishing the same thing, what were they? The purpose of this section is to motivate why the design is the way it is and not some other way.