Status

Current state:  Under Discussion 

Discussion thread: TBD

JIRA: TBD

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

In the admin client (Incremental)AlterConfig API, we implicitly support a feature to reload key/trust store by sending an (Incremental)AlterConfig request directly to the target broker with the exact same store path. This special logic will no longer work once all the AlterConfig requests get forwarded to the active controller, and the target broker will not do a security store reload since the config change ZK notification contains the same key/trust store path as its local copy. In addition, the key/trust store reloading is a completely separate feature from AlterConfigs. It does not change any persistent broker config values in either ZK or metadata quorum. Instead of using client RPCs to directly trigger updates, we propose to use a file watcher on the security store file instead, to listen to any file content change and reload the config as necessary in the post-KIP-500 world.

Public Interfaces

We shall add a broker side metrics to track the reloading the security stores as:

MBean:kafka.server:type=RequestMetrics,name=NumSecurityStoreReloadsPerSec,store_type=([-.\w]+)

where the current supported store types are: key_store|trust_store.

{
  "apiKey": N,
  "type": "response",
  "name": "AlterBrokerPropertyResponse",
  "validVersions": "0",
  "flexibleVersions": "0+",
  "fields": [
    { "name": "BrokerProperties", "type": "[]BrokerProperty",
      "versions": "0+", "fields": [
      { "name": "Key", "type": "string", "versions": "0+",
        "about": "The property key name." },
	  { "name": "ErrorCode", "type": "int16", "versions": "0+",
        "about": "The property alteration error code." },
      { "name": "ErrorMessage", "type": "string", "nullableVersions": "0+", "versions": "0+",
        "about": "The property alteration error message, or null if there was no error." },
      ]
    }
    ]
}

And a new admin client API called `reloadSecurityStore` will be added to do the key/trust store reloading.

/**
 * Reload security stores after new store setup is done.
 * @params options options set to reload the security stores 
 */
ReloadSecurityStoresResult reloadSecurityStores(ReloadSecurityStoresOptions options);

And the store reload options will be available for reloading a set of brokers' store:

Proposed Changes

When AdminClient#reloadSecurityStores is called, one or more AlterBrokerPropertyRequests will be sent to the target brokers for security store reloading.  For older brokers who could not handle this request, the old alterConfig path should still be effective.

For older admin clients, if the AlterConfigRequest gets forwarded the active controller, the controller shall detect the intention for reloading key/trust store configs by looking at their config path value. If the paths are the same, a separate AlterBrokerPropertyRequest shall be sent to the target broker for store reloading. 

Compatibility, Deprecation, and Migration Plan

Rejected Alternatives

We have proposed to use a new RPC which gets sent to the target broker directly. The solution involves a couple of compatibility related concerns, such as the following matrix:

We have attempted a hacky solution by augmenting the security store path from single slash to double slash, like "/foo/bar" to "//foo//bar", in the hope of triggering an equivalent path reloading. This approach does solve the existing problem, but it is error-prone and does not meet a good maintainability standard. For example, it does not take care of the back slash path in Windows platform, also could potentially lead to path explosion if not implemented correctly as "///////foo///////bar". We certainly don't want to maintain such a hacky feature or extra effort to make it understandable to developers.