THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
It doesn't make sense to address direct ZK access in the #1 since connecting to ZooKeeper and applying/removing ZooKeeper ACLs is the whole point of the tool. (In theory we could replace its direct ZK access in favor of a Kafka API, but that seems silly.)
...
Direct ZK access in #3 has already been replaced via a --bootstrap-server flag and will be deprecated in the next release via KIP-555 as well. A comment in the code at https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/ConfigCommand.scala#L65) indicates that connecting directly to ZooKeeper with this CLI tool is still a supported use case when configuration information needs to be bootstrapped into a ZooKeeper quorum prior to starting Kafka. This is a very special use case for sure, but it does mean that accessing a ZooKeeper instance directly via this CLI tool will be required, and passing TLS configuration to it in a secured way will be necessary.
There is an additional CLI tool that supports bootstrapping information into ZooKeeper besides ConfigCommand: kafka-acls.{bat,sh} (kafka.admin.AclCommand). Accessing a ZooKeeper instance directly via this CLI tool will also be required, and passing TLS configuration to it in a secured way will also be necessary.
...
- Harden/secure the configuration mechanism for Zookeeper TLS connectivity from:
- Kafka Brokers (including from
kafka.security.authorizer.AclAuthorizerif/when configured) zookeeper-security-migration.shkafka-configs.{bat,sh}andkafka-acls.{bat,sh}zookeeper-shell.{bat,sh}
- Kafka Brokers (including from
- Support client certificate authentication to ZooKeeper both with and without SASL authentication in ZK Security Migrator and the broker (when
zookeeper.set.aclis true). - Add system tests to confirm the hardened/secured configuration for TLS connectivity to ZooKeeper
- Add explicit Kafka documentation on how to configure TLS connectivity to ZooKeeper
- Add a reference in the Kafka documentation to the ZooKeeper Quorum TLS configuration (https://zookeeper.apache.org/doc/r3.5.6/zookeeperAdmin.html#Communication+using+the+Netty+framework)
...
As an example, these are some of the configs that will be introduced:
zookeeper.ssl.client.secureenable=truezookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNettyzookeeper.ssl.keyStorekeystore.location=kafka.jkszookeeper.ssl.keyStorekeystore.password=test1234zookeeper.ssl.trustStoretruststore.location=truststore.jkszookeeper.ssl.trustStoretruststore.password=test1234
Every config can be prefixed with "authorizer." for the case when kafka.security.authorizer.AclAuthorizer connects via TLS to a ZooKeeper quorum separate from the one that Kafka is using – this specific use case will be identified in the configuration by explicitly setting authorizer.zookeeper.ssl.client.enable=true. In this case the configs prefixed with "authorizer." are not "overrides" like the other authorizer ZooKeeper connectivity configs such as connection/session timeouts and max inflight requests; ZooKeeper TLS connectivity values for the authorizer are not "merged" with Kafka's ZooKeeper TLS configs (if any) because semantically the two sets of configs are for different ZooKeeper quorums and there is no guarantee that they would be applicable across the two quorums; any configs that need to be identical across the two ZooKeeper quorums will have to be repeated with and without the prefix. The same defaults described below The same defaults (if any) described below will apply to the prefixed configs.
| Config Key | Documentation |
|---|---|
| Set client to use TLS when connecting to ZooKeeper. When true, <code>zookeeper.clientCnxnSocket</code> must be set (typically to <code>org.apache.zookeeper.ClientCnxnSocketNetty</code>); other values to set may include <include list of all other properties below> |
zookeeper.clientCnxnSocket | Typically set to <code>org.apache.zookeeper.ClientCnxnSocketNetty</code> when using TLS connectivity to ZooKeeper |
zookeeper.ssl.keystore.location | Keystore location when using a client-side certificate with TLS connectivity to ZooKeeper. Overrides any explicit value set via the <code>zookeeper.ssl.keyStore.location</code> system property (note the camelCase) and inherits the value of <code>ssl.keystore.location</code> if no explicit value is set both here and via the system property. |
| Keystore password when using a client-side certificate with TLS connectivity to ZooKeeper. Overrides any explicit value set via the <code>zookeeper.ssl.keyStore.password</code> system property (note the camelCase) and inherits the value of <code>ssl.keystore.password</code> if no explicit value is set both here and via the system property. |
| Keystore type when using a client-side certificate with TLS connectivity to ZooKeeper. The default value of <code>null</code> means the type will be auto-detected based on the filename extension of the keystore Overrides any explicit value set via the <code>zookeeper.ssl.keyStore.type</code> system property (note the camelCase) and inherits the value of <code>ssl.keystore.type</code> (and its default value if necessary) if no explicit value is set both here and via the system property. |
zookeeper.ssl.truststore.location | Truststore location when using TLS connectivity to ZooKeeper. Overrides any explicit value set via the <code>zookeeper.ssl.trustStore.location</code> system property (note the camelCase) and inherits the value of <code>ssl.truststore.location</code> if no explicit value is set both here and via the system property. |
zookeeperzookeeper.ssl.truststore.password
| Truststore password when using TLS connectivity to ZooKeeper.zookeeper Overrides any explicit value set via the <code>zookeeper.ssl.truststore.typetrustStore.password</code> system property (note the camelCase) and inherits the value of <code>ssl.truststore.password</code> if no explicit value is set both here and via the system property. |
| Truststore type when using TLS connectivity to ZooKeeper. The default value of <code>null</code> means the type will be auto-detected based on the filename extension of the truststore Overrides any explicit value set via the <code>zookeeper.ssl.trustStore.type</code> system property (note the camelCase) and inherits the value of <code>ssl.truststore.type</code> (and its default value if necessary) if no explicit value is set both here and via the system property. |
| Specifies the protocol to be used in ZooKeeper TLS negotiation. Overrides any explicit value set via the <code>zookeeper.ssl.protocol</code> system property and inherits the value of <code>ssl.protocol</code> (and its default value if necessary) if no explicit value is set both here and via the system property. |
zookeeper.zookeeper.ssl.enabled.protocols | Specifies the enabled protocol(s) in ZooKeeper TLS negotiation (csv). The default value of <code>null</code> means the enabled protocol will be the value of the <code>zookeeper.ssl.protocol</code> configuration Overrides any explicit value set via the <code>zookeeper.ssl.enabledProtocols</code> system property (note the camelCase) and inherits the value of <code>ssl.enabled.protocols</code> (and its default value if necessary) if no explicit value is set both here and via the system property. |
zookeeper.ssl.cipher.suites | Specifies the enabled cipher suites to be used in ZooKeeper TLS negotiation (csv). The default value of <code>null</code> means the list of enabled cipher suites is determined by the Java runtime being used Overrides any explicit value set via the <code>zookeeper.ssl.ciphersuites</code> system property (note the single word "ciphersuites") and inherits the value of <code>ssl.cipher.suites</code> (if any) if no explicit value is set both here and via the system property. |
zookeeper.ssl.context.supplier.class | Specifies the class to be used for creating SSL context in ZooKeeper TLS communication |
| Specifies whether to Specifies whether to enable hostname verification in the ZooKeeper TLS negotiation process. Disabling it is only recommended for testing purposesZooKeeper TLS negotiation process, with (case-insensitively) "https" meaning ZooKeeper hostname verification is enabled and an explicit blank value meaning it is disabled (disabling it is only recommended for testing purposes). Overrides any explicit "true" or "false" value set via the <code>zookeeper.ssl.hostnameVerification</code> system property (true implying https and false implying blank) and inherits the value of <code>ssl.endpoint.identification.algorithm</code> (if any) if no explicit value or non-value is set both here and via the system property. |
| Specifies whether to enable Certificate Revocation List in the ZooKeeper TLS protocols |
| Specifies whether to enable Online Certificate Status Protocol in the ZooKeeper TLS protocols |
...
This config file may contain any of the above properties to configure the TLS connection to ZooKeeper; any config(s) not mentioned in the above list will be ignored (a convenience to make it possible to pass a Broker properties file)
...
The same --zk-tls-config-file parameter will be added to support the bootstrap use case.
ACL Command CLI
The same --zk-tls-config-file parameter will be added to support the bootstrap use case.
ZooKeeper Shell CLI
A -zk-tls-config-file parameter will be added. Note the use of single-dash as opposed to double-dash here since all of the tool's parameters follow the ZooKeeper project's style and are specified via a single dasha single dash. The "usage" message will be updated accordingly.
Proposed Changes
The proposed changes include the public interface changes:
- New Kafka configurations, both non-prefixed as well as prefixed with "
authorizer.", with existing Kafka configurations inherited as described. - A new --zk-tls-config-file parameter for:
- ZooKeeper Security Migration Tool
- Config Command CLI (for the special use case of bootstrapping TLS-enabled ZooKeeper)
- ACL Command CLIs (for the special use case of bootstrapping TLS-enabled ZooKeeper)
- A new -zk-tls-config-file parameter in the ZooKeeper Shell (again, note the single dash as opposed to the double-dash used above)
ZooKeeper supports Java system properties for configuration, and there are Kafka TLS configuration keys/values that are semantically equivalent to the ZooKeeper configurations in many case as well, so we will define ; this is described above, but for clarity it is also described as an evaluation algorithm for configuration values as followshere:
- System properties define an initial value to be used. For example, if the Java process is started with "
-Dzookeeper.ssl.trustStore.location=/the/path" then that will be the starting value for thezookeeper.ssl.truststore.locationconfiguration. Note that the system properties use the ZooKeeper camelcase camelCase convention; this cannot be changed. - Explicit configuration values take precedence over any value defined via a system property. So if in addition to the above system property the configuration explicitly includes "
zookeeper.ssl.truststore.location=/another/path" then that will be the value for thezookeeper.ssl.truststore.locationconfiguration - If no value exists at this point – no system property and no explicit value – then any explicit, equivalent Kafka configuration will be used. So, for example, if no such system property in (1) or config in (2) existed but the same config file contained "
ssl.truststore.location=/the/kafka/path" then that would be the value used.
The list of configs for which the above algorithm will be used is as followsbe used is as follows (this is consistent with the configuration descriptions provided in the table above – it is presented here in this form for clarity):
| Zk System Property | "zookeeper." config | Potentially "Inherited" Kafka Config |
|---|---|---|
zookeeper.ssl.keyStore.{location, password,type} | zookeeper.ssl.keystore.{location,password,type} | ssl.keystore.{location,passsord,type} |
zookeeper.ssl.trustStore.{location, password,type} | zookeeper.ssl.truststore.{location,password,type} | ssl.truststore.{location,passsord,type} |
zookeeper.ssl.ciphersuites | zookeeper.ssl.cipher.suites | ssl.cipher.suites |
zookeeper.ssl.protocol | zookeeper.ssl.protocol | ssl.protocol |
zookeeper.ssl.enabledProtocols | zookeeper.ssl.enabled.protocols | ssl.enabled.protocols |
zookeeper.ssl.hostnameVerification | zookeeper.ssl.endpoint.identification.algorithm | zssl.endpoint.identification.algorithm |
Note that inheritance occurs on brokers and on CLI tools.Note that inheritance of Kafka configs does not occur , and it also occurs for AclAuthorizer on the broker when it is explicitly pointed at a separate ZooKeeper quorum , but system properties will be applied in this case anyway (these cannot be disabled).(with all explicit ZooKeeper configs in the second column starting with "authorizer.zookeeper."
The proposed changes also include the addition of:
...
The changes are additions only, and there is no compatibility issue in the broker because the default for the broker config zookeeper.client.secure is false. TLS to ZooKeeper is an opt-in even when TLS is enabled between brokers because zookeeper.client.secure doesn't inherit its value from anywhere and defaults to false unless explicitly set to true.
Test Plan
System tests will cover the following:
...
ZooKeeper uses camelCase configs that are inconsistent with Kafka broker configs:
| ZooKeeper Config | Kafka Broker Config |
|---|---|
| zookeeper.ssl.keyStore.location | ssl.keystore.location |
| zookeeper.ssl.keyStore.password | ssl.keystore.password |
| zookeeper.ssl.keyStore.type | ssl.keystore.type |
| zookeeper.ssl.trustStore.location | ssl.truststore.location |
| zookeeper.ssl.trustStore.password | ssl.truststore.password |
| zookeeper.ssl.trustStore.type | ssl.truststore.type |
| zookeeper.ssl.ciphersuites | ssl.cipher.suites |
| zookeeper.ssl.enabledProtocols | ssl.enabled.protocols |
It would be confusing and prone to mistake to have such a mismatch – especially for people who tend to know very little about ZooKeeper compared to Kafka.
The ZooKeeper config "zookeeper.client.secure" is also confusing – it only refers to whether the client should be configured to communicate with a TLS-encrypted ZK socket and has nothing to do with any other security-related client configuration that might be implied by the term "secure" (e.g. SASL, ACLs). We therefore use the broker-side config "zookeeper.ssl.client.enable" instead.