...

This proposal intends to enhance the AbstractConfig base class to automatically resolve variables of the form specified in KIP-297. This simple change will make it very easy for client applications, Kafka Connect, and Kafka Streams to use shared code to easily incorporate externalized secrets and other variable replacements within their configurations. This proposal does not add other ConfigProvider implementations or change the behavior of existing methods.

...

Public Interfaces

This proposal will add 2 new constructors to interfaces for the AbstractConfig class:
AbstractConfig

...

In order to support components which do not have configProviders defined in the config file but read it from a separate config provider file, we add another constructor with an additional field to specify the config providers Map<?, ?> configProviders. This constructor will first instantiate the ConfigProviders using the map of config provider and then resolve the variable as previous constructor. Any providers in originals map will be ignored. 

Configuration Changes

This KIP proposes to reserve all config names starting with "config.providers" to resolve the config values. The config "config.providers"  will be reserved to list the ConfigProviders, "config.providers.providerName.class" will be reserved to specify the class to be used to instantiate the providers and  "config.providers.providerName.variableName" to specify any additional configs required by providers. The new constructor will look for property for "config.providers" for the list of ConfigProviders to be used for indirect variable resolution. in the originals fields and instantiate them using the specified configProvider class. It will then use these instantiated config providers to resolve any indirect values.   

Proposed Changes 

Example

The following is an example config file that defines the configs and config providers in the same file and another notional ConfigProvider implementation named "vault":
Example of ConfigValues

The first foo.baz property is a typical name-value pair commonly used in all Kafka configuration files. The foo.bar property has a value that is a KIP-297 variable of the form "${providerName:[path:]key}", where "providerName" is the name of a ConfigProvider, "path" is an optional string, and "key" is a required string. Per KIP-297, this variable is resolved by passing the "foo.bar" key and "/path/to/variables.properties" path to a ConfigProvider with the name "file", and the string returned from the ConfigProvider is then used in place of the variable.

An application or component can use or create a custom subclass of the AbstractConfig, and use this class to parse and validate the configurations in a configuration file. If the application or component does not provide any ConfigProviders, then the AbstractConfig class will simply parse the configuration as before without resolving the variable. If, however, the component or application does pass a config properties for ConfigProviders to the AbstractConfig constructor, the constructor will attempt to instantiate the ConfigProvider instance and use it to resolve and replace the foo.bar variable value prior to parsing and validating the configuration.  

Example use of the new AbstractConfig constructor

...

This new constructor will first instantiate the FileConfigProvider using the props configuration properties, and then attempt to resolve the ${file:/path/to/variables.properties:foo.bar} variable with the FileConfigProvider instance by making a call similar to fileConfigProvider.get("/path/to/variables.properties", Collections.singletonSet("foo.bar"), and using the result as the new value for the foo.bar property in the configuration. The constructor will then parse and validate the configuration using the supplied ConfigDef as normal.

Example of the new AbstractConfig constructor with separate configProvider

...