...

We shall add a broker side metrics to track the successful reloading the security stores as:

MBean:kafka.server:type=RequestMetricsBrokerMetrics,name=NumSecurityStoreReloadsPerSecNumSecurityStoreReloadSuccessPerSec,store_type=([-.\w]+)

where the current supported store types are: key_store|trust_store.

{
  "apiKey": N,
  "type": "response",
  "name": "AlterBrokerPropertyResponse",
  "validVersions": "0",
  "flexibleVersions": "0+",
  "fields": [
    { "name": "BrokerProperties", "type": "[]BrokerProperty",
      "versions": "0+", "fields": [
      { "name": "Key", "type": "string", "versions": "0+",
        "about": "The property key name." },
	  { "name": "ErrorCode", "type": "int16", "versions": "0+",
        "about": "The property alteration error code." },
      { "name": "ErrorMessage", "type": "string", "nullableVersions": "0+", "versions": "0+",
        "about": "The property alteration error message, or null if there was no error." },
      ]
    }
    ]
}

And a new admin client API called `reloadSecurityStore` will be added to do the key/trust store reloading.

/**
 * Reload security stores after new store setup is done.
 * @params options options set to reload the security stores 
 */
ReloadSecurityStoresResult reloadSecurityStores(ReloadSecurityStoresOptions options);

And the store reload options will be available for reloading a set of brokers' store:

Proposed Changes

When AdminClient#reloadSecurityStores is called, one or more AlterBrokerPropertyRequests will be sent to the target brokers for security store reloading.  For older brokers who could not handle this request, the old alterConfig path should still be effective.

For older admin clients, if the AlterConfigRequest gets forwarded the active controller, the controller shall detect the intention for reloading key/trust store configs by looking at their config path value. If the paths are the same, a separate AlterBrokerPropertyRequest shall be sent to the target broker for store reloading. 

...

Proposed Changes

Once the editing for the security store file was done, user should expect the above metric to reflect the reloading of security store instantly. We would also add INFO level message to indicate a reloading event has been triggered, with the reload results following.

Compatibility, Deprecation, and Migration Plan

The feature support for same name store reloading will be working in the new brokers automatically. We would try to communicate the deprecation of the old AlterConfig based path in the broker WARN log to let user read the metric instead to evaluate whether the security store is reloaded successfully. 

Rejected Alternatives

We have proposed to use a new RPC which gets sent to the target broker directly. The solution involves a couple of compatibility related concerns, such as the following matrix:

We have also attempted a hacky solution by augmenting the security store path from single slash to double slash, like "/foo/bar" to "//foo//bar", in the hope of triggering an equivalent path reloading. This approach does solve the existing problem, but it is error-prone and does not meet a good maintainability standard. For example, it does not take care of the back slash path in Windows platform, also could potentially lead to path explosion if not implemented correctly as "///////foo///////bar". We certainly don't want to maintain such a hacky feature or extra effort to make it understandable to developers.