...

Kafka's ConfigDef is used for specifying the set of expected configurations throughout the clients, Kafka Connect, and Kafka Streams. For each configuration, you can specify the name, the type, the default value, the documentation, the group information, the order in the group, the width of the configuration value, the name suitable for display in the UI, validation logic the user may provide to perform single configuration validation, recommenders to get valid values for a configuration given the current configuration values etc. Kafka's AbstractConfig class is a convenient base class for components to extend and use to parse, validate, and access configurations that adhere to a specified ConfigDef. The AbstractConfig class holds both the original configuration that was provided as well as the parsed configurations.

KIP-297 was recently introduce introduced to provide a way to use variables within configuration files and a ConfigProvider extension to replace these variables with values found outside the configuration file. This mechanism allows Connect, for example, to store secrets outside of a connector configuration and to insert those externally stored secrets into the configuration just prior to usage. However, components that use the KIP-297 ConfigProviders must manage those providers and explicitly replace the variables.

This proposal intends to enhance the AbstractConfig base class to automatically resolve variables of the form specified in KIP-297. This simple change will make it very easy for all the components client applications, Kafka Connect, and Kafka Streams to use shared code to easily incorporate externalized secrets and other variable replacements within their configurations. This KIP will enable all All the components broker, connect, producer, consumer, admin client, and so forth.  will be able to automatically resolve the external configurations. This proposal does not add other ConfigProvider implementations or change the behavior of existing methods. By default this feature will be enabled only for the Connect component and other components can enable the feature as we will enable the automatic resolution of externalized configurations, each component can disable the feature if needed.

Public Interfaces

This proposal will add 2 new interfaces for the AbstractConfig class:
AbstractConfig

...

The new constructor is similar to an existing constructor, with a flag enableExternalConfigResolution to enable/disable automatic resolution of indirect variablevariables. The originals configurations will contain both the config provider configs as well as configuration properties. The constructor will first instantiate the ConfigProviders using the config provider configs, then it will find all the variables in the values of the originals configurations, attempt to resolve the variables using the named ConfigProviders, and then do the normal parsing and validation of the configurations. If ConfigProvider is not provided in the originals, the constructor will skip the variable substitution step and will simply validate and parse the supplied configuration. With this approach both the configs and config providers can be defined in the same config property file. By default the enableExternalConfigResolution will be set to falsetrue.

Variables are defined by KIP-297 and have the form "${providerName:[path:]key}", where "providerName" is the name of a ConfigProvider, "path" is an optional string, and "key" is a required string. Per KIP-297, this variable is resolved by passing the "key" and optional "path" to a ConfigProvider with the specified name, and the result from the ConfigProvider is then used in place of the variable. Variables that cannot be resolved by the AbstractConfig constructor will be left unchanged in the configuration.

The second constructor provides a way to support components which do not have configProviders defined in the config file but read it from a separate config provider file.  This constructor takes an additional field to specify the config providers Map<?, ?> configProviders. This constructor will first instantiate the ConfigProviders using the map of config provider and then resolve the variable as previous constructor. Any providers in originals map will be ignored. By default the enableExternalConfigResolution will be set to falsetrue.

Configuration Changes

This KIP proposes to reserve all config names starting with "config.providers" to resolve the config values. The config "config.providers"  will be reserved to list the ConfigProviders, "config.providers.<providerName>.class" will be reserved to specify the class to be used to instantiate the providers and  "config.providers.<providerName>.<variableName>" to specify any additional configs required by providers where <providerName> is the name of the new config provider and <variableName> are the variables required by the config provider. The new constructor will look for "config.providers" in the originals fields and instantiate them using the specified configProvider class. It will then use these instantiated config providers to resolve any indirect values.   

Proposed Changes 

Steps to enable the automatic resolution of externalized config feature

This KIP will enable With this KIP all the components broker, connect, producer, consumer, admin client, and so forth.  to will be able to automatically resolve the external configurations. As this feature is built upon KIP-297, each component will have to make the below changes in order to enable this featureto add externalized configurations

1. Identify the configs which need to be stored in external systems.
2. Implement a ConfigProvider which will fetch the config value from external system
3. Replace the configs values with variables as defined by KIP-297
4. Enable automatic config resolution by setting the  enableExternalConfigResolution to true in the subclass of AbstractConfig.

Once these variables are added the components will be automatically get the resolved configurations. As Connect component already supports variables in connector configs via KIP 297 it will use this feature to resolve variables in the worker configurations via the existing 'StandaloneConfig' and 'DistributedConfig' subclasses of AbstractConfig.

Dynamic configurations for Brokers

The In order to support automatic resolution of indirect dynamic configurations in , the brokers should be updated will have to reload the configurations when the config values are updated. This can be achieved  by updating the broker using the AdminClient and kafka.sh script as described by KIP-226. The dynamic config updates need to happen atomically in order to achieve this the user will have to first update the external configs after all the external configs are updated, he will send a AdminClient request to the broker to update those config from config provider. Upon receiving the alterConfig request, the broker will refetch the updated configs from the ConfigProvider. The actual resolution of indirect values will be done by the KafkaConfig which extends from AbstractConfig, it will invoke the ConfigProvider to provide the actual values for the updated configs.

...

Compatibility, Deprecation, and Migration Plan

• For this KIP we will be enabling the auto resolution only for Connect component and passing false for the other components.  
• This KIP proposes to reserve all config names starting with "config.providers" to resolve the config values. The config "config.providers"  will be reserved to list the ConfigProviders, "config.providers.<providerName>.class" will be reserved to specify the class to be used to instantiate the providers and  "config.providers.<providerName>.<variableName>" to specify any additional configs required by providers.  If any current configs match this custom configs format and auto resolution is enabled, we will try to resolve the value to a actual value by fetching it from the config provider, If a matching key is found in the config provider it will replace its value and could result in a failure. In order to prevent such side effects it would be recommended to make sure that we will update any existing configs matching the format are updated before enabling auto resolution. 

Rejected Alternatives

Several other designs were considered but ultimately rejected.

...