Status

Current state:  Under Discussion   Approved

Discussion thread: TBD https://lists.apache.org/thread.html/ra490809bd850159fe4f9c4668c2ecf84cf9f4a28d50a4a461d212e72%40%3Cdev.kafka.apache.org%3E

JIRA: https://issues.apache.org/jira/browse/KAFKA-10345

...

In the admin client (Incremental)AlterConfig API, we implicitly support a feature to reload key/trust store by sending an (Incremental)AlterConfig request directly to the target broker with the exact same store path. This special logic will no longer work once all the AlterConfig requests get forwarded to the active controller, and the target broker will not do a security store reload since the config change ZK notification contains the same key/trust store path as its local copy. In addition, the key/trust store reloading is a completely separate feature from AlterConfigs. It does not change any persistent broker config values in either ZK or metadata quorum. Instead of using client RPCs to directly trigger updates, we propose to use a file watcher on the security store file instead, to listen to any file content change and reload the config as necessary in the post-KIP-500 world. To protect the worst case where file-watch does not trigger properly, a time-based reloading mechanism will also be added.

Public Interfaces

We would enforce a file-watch based reloading mechanism to the following configs:

• ssl.keystore.location

• ssl.truststore.location 

A shall add two broker side metrics will be added to track times of security store reloads for success and failure as:

...

where the current supported store types are: key_store|trust_store.

Additionally, a two dynamic broker config called `security.storecalled `ssl.keystore.location.refresh.windowinterval.ms` will and `ssl.truststore.location.refresh.interval.ms` will be added to control the time based guarantee for an automatic reloading in case of a missed file-watch. . To disable periodical reloading, users could set those configs to max long.

public static final String SECURITYSSL_KEYSTORE_STORELOCATION_REFRESH_WINDOWINTERVAL_MS_CONFIG = "securityssl.keystore.storelocation.refresh.windowinterval.ms";
public static final String SECURITYSSL_KEYSTORE_STORELOCATION_REFRESH_WINDOWINTERVAL_MS_DOC = "The refresh interval for in-place ssl keystore updates. In general, " +
   "the update should trigger immediately when user modifies the security file path through file watch service, while " +
   "this configuration is defining a time based guarantee of store reloading in worst case";

public static final String SSL_TRUSTSTORE_LOCATION_REFRESH_INTERVAL_MS_CONFIG = "ssl.truststore.location.refresh.interval.ms";
public static final String SSL_TRUSTSTORE_LOCATION_REFRESH_INTERVAL_MS_DOC = "The refresh interval for in-place ssl truststore updates. In general, " +
   "the update should trigger immediately when user modifies the security file path through file watch service, while " +
   "this configuration is defining a time based guarantee of store reloading in worst case";

The default value values will be set to 5 minutes and could be changed through AlterConfig API.

...

The broker will also periodically check whether security files have been modified since last checking time and decide to reload or not, so that when the file watch does not work for unknown reason, user could wait until the time based reloading mechanism kicks in. If the time based reloading is not working either, user could still try to change the store path in an explicit AlterConfig call in the worst case. 

This mechanism will be applied only to the brokers. Client side security store reloading mechanism is not affected.

Compatibility, Deprecation, and Migration Plan

The feature support for same name store reloading will be working in the new brokers automatically. We would try to communicate the deprecation of the old AlterConfig based path in the broker WARN log to let user read the metric instead to evaluate whether the security store is reloaded successfully. 

Right now we don't plan to support disabling the auto reloading feature, since it will become the only option to do in-place security store update in the future.

We are also changing the audit log/authorization model for dynamic in-place updates of SSL stores. At the moment, only a user with powerful Cluster:Alter permissions can dynamically
update SSL stores on brokers. The KIP removes this restriction and relies purely on file system permissions for file-based stores.

Rejected Alternatives

We have proposed to use a new RPC which gets sent to the target broker directly. The solution involves a couple of compatibility related concerns, such as the following matrix suggests:

...