Versions Compared

Old Version 8

changes.mady.by.user TEJAL ADSUL

Saved on

compared with

New Version 9

changes.mady.by.user TEJAL ADSUL

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

This KIP proposes to reserve all config names starting with "config.providers" to resolve the config values. The config "config.providers"  will be reserved to list the ConfigProviders, "config.providers.providerName.class" will be reserved to specify the class to be used to instantiate the providers and  "config.providers.providerName.variableName" to specify any additional configs required by providers. The new constructor will look for "config.providers" in the originals fields and instantiate them using the specified configProvider class. It will then use these instantiated config providers to resolve any indirect values.   

Dynamic configurations for Brokers
The indirect dynamic configurations in the brokers should be updated using the AdminClient and kafka.sh script as described by KIP-226. The dynamic config updates need to happen atomically in order to achieve this the user will have to first update the external configs after all the external configs are updated, he will send a AdminClient request to the broker to update those config from config provider. Upon receiving the alterConfig request, the broker will refetch the updated configs from the ConfigProvider. The actual resolution of indirect values will be done by the KafkaConfig which extends from AbstractConfig, it will invoke the ConfigProvider to provide the actual values for the updated configs.

Example:

Consider the dynamic configurations ssl.keystore.location, ssl.keystore.password and ssl.keystore.type are stored in vault and their values are resolved using a VaultConfigProvider. 

...
ssl.keystore.location=${vault:/path/to/variables.properties:ssl.keystore.location}

ssl.keystore.password=${vault:/path/to/variables.properties:ssl.keystore.password}

ssl.keystore.type=${vault:/path/to/variables.properties:ssl.keystore.type}
 
config.providers=vault
config.providers.file.class=org.apache.kafka.connect.configs.VaultConfigProvider

  • The user will update the ssl.keystore.location, ssl.keystore.password, ssl.keystore.type in the vault.
  • After the updates are complete he will send a adminClient request to the broker to notify the configs are updated.
  • Broker receives a alteredConfig request.
  • The broker will invoke the get function in VaultConfigProvider
  • The VaultConfigProvider will fetch the actual values for ssl.keystore.location, ssl.keystore.password, ssl.keystore.type from the vault.
  • The broker will then validate the configs and apply the changes.

...

Compatibility, Deprecation, and Migration Plan

...

  • For this KIP we will be enabling the auto resolution only for Connect component and passing false for the other components.

...

  •   
  • This KIP proposes to reserve all config names starting with "config.providers" to resolve the config values. The config "config.providers"  will be reserved to list the ConfigProviders, "config.providers.providerName.class" will be reserved to specify the class to be used to instantiate the providers and  "config.providers.providerName.variableName" to specify any additional configs required by providers.  If any current configs match this custom configs format and auto resolution is enabled, we will try to resolve the value to a actual value by fetching it from the config provider, If a matching key is found in the config provider it will replace its value and could result in a failure. In order to prevent such side effects it would be recommended to make sure that existing configs matching the format are updated before enabling auto resolution. 

Future Enhancement 

...

Rejected Alternatives

Several other designs were considered but ultimately rejected.

...