...

In the admin client (Incremental)AlterConfig API, we implicitly support a feature to reload key/trust store by sending an (Incremental)AlterConfig request directly to the target broker with the exact same store path. This special logic will no longer work once all the AlterConfig requests get forwarded to the active controller, and the target broker will not do a security store reload since the config change ZK notification contains the same key/trust store path as its local copy. In addition, the key/trust store reloading is a completely separate feature from AlterConfigs. It does not change any persistent broker config values in either ZK or metadata quorum. Instead of using client RPCs to directly trigger updates, we propose to use a file watcher on the security store file instead, to listen to any file content change and reload the config as necessary in the post-KIP-500 world. To protect the worst case where file-watch does not trigger properly, a time-based reloading mechanism will also be added.

Public Interfaces

We shall add two broker side metrics to track times of security store reloads for success and failure as:

...

where the current supported store types are: key_store|trust_store.

Additionally, a dynamic broker config called `security.store.refresh.window.ms` will be added to control the time based guarantee for an automatic reloading in case of a missed file-watch. 

public static final String SECURITY_STORE_REFRESH_WINDOW_MS_CONFIG = "security.store.refresh.window.ms";
public static final String SECURITY_STORE_REFRESH_WINDOW_MS_DOC = "The refresh interval for in-place security store updates. In general, " +
   "the update should trigger immediately when user modifies the security file path through file watch service, while " +
   "this configuration is defining a time based guarantee of store reloading in worst case";

The default value will be set to 5 minutes and could be changed through AlterConfig API.

Proposed Changes

Once the editing for the security store file was done, user should expect the above metric to reflect the reloading of security store instantly. We would also add INFO level message to indicate a reloading event has been triggered, with the reload results. If the result is a failure, we would increment the failed metrics and log the reason in ERROR. When the reloading fails, previous store should still be effective.

When The broker will also periodically check whether security files have been modified since last checking time and decide to reload or not, so that when the file watch does not work for unknown reason, user could wait until the time based reloading mechanism kicks in. If the time based reloading is not working either, user could still try to change the store path in an explicit AlterConfig call in the worst case. 

...