Page History

...

The new constructor is similar to an existing constructor, with a flag enableExternalConfigResolution to enable/disable automatic resolution of KIP proposes on using the  existing AbstractConfig to automatically resolve indirect variables. The originals configurations will contain both the config provider configs as well as configuration properties. The constructor will first instantiate the ConfigProviders using the config provider configs, then it will find all the variables in the values of the originals configurations, attempt to resolve the variables using the named ConfigProviders, and then do the normal parsing and validation of the configurations. If ConfigProvider is not provided in the originals, the constructor will skip the variable substitution step and will simply validate and parse the supplied configuration. With this approach both the configs and config providers can be defined in the same config property file. By default the enableExternalConfigResolution will be set to true.

Variables are defined by KIP-297 and have the form "${providerName:[path:]key}", where "providerName" is the name of a ConfigProvider, "path" is an optional string, and "key" is a required string. Per KIP-297, this variable is resolved by passing the "key" and optional "path" to a ConfigProvider with the specified name, and the result from the ConfigProvider is then used in place of the variable. Variables that cannot be resolved by the AbstractConfig constructor will be left unchanged in the configuration.

The second constructor provides We propose to add new constructor which is similar to the existing constructor with an additional field  Map<?, ?> configProviders to specify the config providers to use for resolving the configs. This constructor will provide a way to support components which do not have configProviders defined in the config file but read it from a separate config provider file.   This constructor takes an additional field to specify the config providers Map<?, ?> configProviders. This constructor will first instantiate the ConfigProviders using the map of config provider and then resolve the variable as previous constructor. Any providers in originals map will be ignored. By default the enableExternalConfigResolution will be set to true. 

Configuration Changes

This KIP proposes to reserve all config names starting with "config.providers" to resolve the config values. The config "config.providers"  will be reserved to list the ConfigProviders, "config.providers.<providerName>.class" will be reserved to specify the class to be used to instantiate the providers and  "config.providers.<providerName>.<variableName>" to specify any additional configs required by providers where <providerName> is the name of the new config provider and <variableName> are the variables required by the config provider. The new constructor will look for "config.providers" in the originals fields and instantiate them using the specified configProvider class. It will then use these instantiated config providers to resolve any indirect values.   

...

An application or component can use or create a custom subclass of the AbstractConfig, and use this class to parse and validate the configurations in a configuration file. If the application or component does not provide any ConfigProviders, then the AbstractConfig class will simply parse the configuration as before without resolving the variable. If, however, the component or application does pass a config properties for ConfigProviders to the AbstractConfig constructor, the constructor will attempt to instantiate the ConfigProvider instance and use it to resolve and replace the foo.bar variable value prior to parsing and validating the configuration.  

Example use of the new existing AbstractConfig constructor

This new constructor will first instantiate the FileConfigProvider using the props configuration properties, and then attempt to resolve the ${file:/path/to/variables.properties:foo.bar} variable with the FileConfigProvider instance by making a call similar to fileConfigProvider.get("/path/to/variables.properties", Collections.singletonSet("foo.bar"), and using the result as the new value for the foo.bar property in the configuration. The constructor will then parse and validate the configuration using the supplied ConfigDef as normal.

...

This new constructor will first instantiate the ConfigProviders passed in the configProviders object and then look for any variables in the supplied props object, find the ${file:/path/to/variables.properties:foo.bar} variable and attempt to resolve it with the FileConfigProvider instance by making a call similar to fileConfigProvider.get("/path/to/variables.properties", Collections.singletonSet("foo.bar"), and using the result as the new value for the foo.bar property in the configuration. If any configProviders are passed in props object they will be ignored. The constructor will then parse and validate the configuration using the supplied ConfigDef as normal. 

...

1. Assume the ConfigProvider instances are also configured in the same configuration provided to the AbstractConfig constructor. This was rejected because it did not provide enough flexibility. For example, in Kafka Connect, the worker configurations can define ConfigProviders, whereas the connector configurations do not. Assuming the ConfigProviders are defined in the connector configuration would change the behavior of Connect.
2. Place the new logic in a separate class or as static methods. This is a viable option, but the current design was thought to be a bit more usable. 
3. In order to support dynamic configs add a new DynamicConfiguration class that allows a component to obtain an initial configuration instance (subclass of AbstractConfig) and to add a listener that will be called when any of the configuration properties resolved from ConfigProviders change in the underlying provider. This was rejected as it does not handle atomic updates efficiently which is required by broker configs. 
4. In order to support atomic dynamic configs enhance ConfigProvider to handle batch updates of configs by adding few seconds of delay before the fetching the actual values  and modify ConfigChangeCallback to pass all the config changes at once in one giant map. This is not a viable option as related configs may be obtained from different ConfigProviders and adding delay in the config provider won't guarantee atomic updates.
5. By default enable automatic resolution of externalized configs in AbstractConfig constructorAdd a new constructor to AbstractConfig with a flag to enable/disable the feature by the subclass of AbstractConfig. This approach was rejected as it gives more flexibility the user won't be able to enable/disable the feature using configs or cli and feature by the subclass of AbstractConfig should be enabled by default for all the components.