Status

Current state: "Under Discussion"Adopted" (in version 2.5)

Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]

JIRA:

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

ZooKeeper TLS Functionality

Since the merge KAFKA-8634 (https://github.com/apache/kafka/commit/d67495d6a7f4c5f7e8736a25d6a11a1c1bef8d87)   in trunk, Apache Kafka can work ships with the latest Apache Zookeeper supporting TLS and Dynamic Reconfiguration (AK 2.4 ultimately shipped with ZooKeeper version 3.5.6 rather than 3.5. 

This version comes with two very expected features, the support for TLS connections and the introduction of Dynamic Reconfiguration.

5, but the general functionality is the same).  When doing a deployment in a security-minded environment the desire is to use TLS to encrypt communication in transit.

Note that the current version of ZooKeeper (3.5.6 as of this writing and the version shipped with Apache Kafka 2.4) only supports mutual certificate authentication.  There is a sever-side config "ssl.clientAuth" that the ZooKeeper code recognizes (case-insensitively: want/need/none are the valid options), but this config has no effect in 3.5.6 (

Note also that ZooKeeper will associate multiple identities with any session that successfully authenticates multiple ways (e.g. both client certificate and SASL).  The X.509 identity is the full Distinguished Name from the client's certificate, and this can be changed (i.e. use just a part of the DN) only by implementing and using a custom ZooKeeper authentication provider that overrides the method protected String getClientId(X509Certificate clientCert). A client that accesses an ACL-protected Znode is authorized if it has at least 1 of the identities present in any authorizing ACL.

ZooKeeper also supports TLS connectivity between ZK nodes for Quorum-related communication.  This is configured independent of Kafka within ZooKeeper.

It is possible to enable TLS connectivity to Zookeeper from Apache Kafka 2.4 -- the problem is that configuration information has to be passed via system properties as -D command line options on the Java invocation of the broker or CLI tool (e.g. ZooKeeper Security Migration), and such -D  command line options are not secure because anyone with access to the box can see the command line used to start the process; the configuration includes sensitive keystore/truststore password information, so we need a secure mechanism for passing the configuration values. The motivation for this KIP is to harden/secure the configuration mechanism for Zookeeper TLS connectivityThe current internal zookeeper client version, at the time of this KIP, support JAAS, SASL, based communication credentials. When doing a deployment in a security minded environment the user aim to use TLS to secure encryption in transit.

With this KIP we aim to introduce the necessary changes to enable the possibility to use a use of secure configuration values when defining TLS encrypted channel channels for communications with Zookeeper. These changes will enable the secure use of TLS with the Zookeeper Security Migration CLI as well within the internal usages.

Goals

• Extend the internal zookeeper client to get external configuration from a file.
• Extend the Zookeeper Security Migration CLI to allow passing a property.
• Add support for the necessary zookeeper SSL configuration variables within the server.properties file.

Public Interfaces

Zookeeper Security Migration CLI 

The existing configuration option java.security.auth.login.config, used to set provide the JAAS configuration content will be kept as it is. 

A new parameter will be added:

• zookeeper.config.path: This CLI parameter will allow users to pass a zookeeper properties file with the relevant configuration to be used to connect with the zookeeper server.

The zookeeper.config.path parameter will take precedence over the java.security.auth.login.config option.

Config Command CLI 

Still some other CLIs have the option have the option to connect to ZK. This commands still support existing java.security.auth.login.config to support a SASL/JAAS based authentication. For the purpose of this KIP a new parameter will be added:

• zookeeper.config.path: This CLI parameter will allow users to pass a zookeeper properties file with the relevant configuration to be used to connect with the zookeeper server.

The zookeeper.config.path parameter will take precedence over the java.security.auth.login.config option.

Other command that will undergo the same transformation are:

• PreferredReplicaLeaderElectionCommand
• TopicCommand
• ReassignPartitionsCommand

Kafka Configuration file (server.properties)

from brokers as well as any CLI tools that require it in the next AK (AK 2.5) release.

Brokers

Brokers talk to ZooKeeper, of course.  In addition, the class kafka.security.authorizer.AclAuthorizer talks directly to ZooKeeper and supports being pointed to a separate ZooKeeper quorum, so it must be possible to configure that ZooKeeper connection for TLS as well.  Note that kafka.security.auth.SimpleAclAuthorizer was deprecated in AK 2.4 (in favor of AclAuthorizer) and will not support TLS connectivity to ZooKeeper.

CLI Tools

The list of CLI tools that used non-deprecated direct ZooKeeper access in the previous AK (AK 2.4) release was as follows:

1. zookeeper-security-migration.sh (kafka.admin.ZSecurityMigrator)
2. kafka-reassign-partitions.{bat,sh} (kafka.admin.ReassignPartitionsCommand)
3. kafka-configs.{bat,sh} (kafka.admin.ConfigCommand)
4. zookeeper-shell.{bat,sh}

It doesn't make sense to address direct ZK access in #1 since connecting to ZooKeeper and applying/removing ZooKeeper ACLs is the whole point of the tool.  (In theory we could replace its direct ZK access in favor of a Kafka API, but that seems silly.)

Direct ZK access in #2 above is being addressed via the already-accepted KIP-455: Create an Administrative API for Replica Reassignment, and the direct access flag will be deprecated via KIP-555: Deprecate Direct Zookeeper access in Kafka Administrative Tools.  Therefore no changes are required here.

Direct ZK access in #3 has already been replaced via a --bootstrap-server flag and will be deprecated in the next release via KIP-555 as well. A comment in the code at https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/ConfigCommand.scala#L65) indicates that connecting directly to ZooKeeper with this CLI tool is still a supported use case when configuration information needs to be bootstrapped into a ZooKeeper quorum prior to starting Kafka.  This is a very special use case for sure, but it does mean that accessing ZooKeeper directly via this CLI tool will be required, and passing TLS configuration to it in a secured way will be necessary.

Direct ZooKeeper access is the whole point of #4, of course, but the ZooKeeper project does not yet provide functionality to pass secured TLS configs to the underlying class that we invoke from this shell script – org.apache.zookeper.ZooKeeperMain – as described in 

kafka-acls.{bat,sh} (kafka.admin.AclCommand) is an additional CLI tool not in the above list that supports bootstrapping information into ZooKeeper.  Direct ZooKeeper access has been deprecated in the ACLs CLI tool for some time, but it is still required for this special use case, and passing TLS configuration in a secured way will be necessary.

Goals

• Harden/secure the configuration mechanism for Zookeeper TLS connectivity from:
  • Kafka Brokers (including from kafka.security.authorizer.AclAuthorizer if/when configured)
  • zookeeper-security-migration.sh
  • kafka-configs.{bat,sh} and kafka-acls.{bat,sh}
    
  • zookeeper-shell.{bat,sh}
• Support client certificate authentication to ZooKeeper both with and without SASL authentication in ZK Security Migrator and the broker (when zookeeper.set.acl is true).
• Add system tests to confirm the hardened/secured configuration for TLS connectivity to ZooKeeper
• Add explicit Kafka documentation on how to configure TLS connectivity to ZooKeeper – both mutual TLS and encryption-only
• Add a reference in the Kafka documentation to the ZooKeeper Quorum TLS configuration (https://zookeeper.apache.org/doc/r3.5.7/zookeeperAdmin.html#Communication+using+the+Netty+framework)

Out of Scope

• Zookeeper-to-Zookeeper Quorum TLS system tests and in-depth documentation (the ZooKeeper project already has such tests and documentation)
• Dynamic reconfiguration of ZooKeeper TLS configs

Public Interfaces

New Broker and AclAuthorizer Configurations

The below table contains the complete list of added configs.  All configs being added are optional Strings with no default value unless otherwise noted.  These values are potentially required to access ZooKeeper in the first place, so they are not dynamically reconfigurable (dynamic reconfiguration values are currently stored in ZooKeeper).  Sensitive values (e.g. those of type Password) can be encrypted as described in KIP-421: Automatically resolve external configurations.

As an example, these are some of the configs that will be introduced:

zookeeper.ssl.client.enable=trueNew configuration properties will be added to the KafkaConfig object. As an example, these are the ones that will be introduced:
zookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
zookeeper

...

.

...

ssl.

...

keystore.location=kafka.jks
zookeeper.ssl.

...

keystore.password=test1234
zookeeper.ssl.

...

truststore.location=truststore.jks
zookeeper.ssl.

...

truststore.password=test1234

Simple ACL Authorizer configuration (server.properties)

Currently the Simple ACL authorizer had the option to use different zookeeper connection settings. With this KPI we will need to add the variables need to connect with TLS. 

With this KPI we will add this variables:

authorizer.zookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
authorizer.zookeeper.client.secure=true
authorizer.zookeeper.ssl.keyStore.location=kafka.jks
authorizer.zookeeper.ssl.keyStore.password=test1234
authorizer.zookeeper.ssl.trustStore.location=truststore.jks
authorizer.zookeeper.ssl.trustStore.password=test1234

Proposed Changes

CLI tools

Each CLI tool that still allow to use zookeeper like:

• PreferredReplicaLeaderElectionCommand
• TopicCommand
• ReassignPartitionsCommand
• ConfigCommand
• Zookeeper Security Migration

Will be adapted to use a config file as a compliment to the current JAAS file method.

KafkaConfiguration

Several new variables, required to have a TLS connection with zookeeper will be added to the configuration definition. These variables will be passed across and used internally when necessary during the creation of the KafkaZkClient objects.

KafkaZkClient

Extend the apply method to hold a Map[String, Any] to support passing any number of configuration parameters being passed to the low level Zk client.

Compatibility, Deprecation, and Migration Plan

The changes are planned to be introduced in a compatible way, by keeping the current JAAS variable precedence. Current users will not see any change, but if a user would like to use a zookeeper server with a TLS connection, it will be able to pass a config file, or update the server.properties file.

Test Plan

A new integration test will be adapted to use the new configurations for TLS connection with Zookeeper. If need relevant existing integration and system tests will be updated accordingly.

Rejected Alternatives

Use the global system variables as proposed in the Zookeeper User guide

The Zookeeper user guide (https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZooKeeper+SSL+User+Guide) proposes the usage of two environment variables, CLIENT_JVMFLAGS and SERVER_JVMFLAGS, to configure the usage of TLS in the client and server zookeeper sides.

This option has been considered and rejected, while it could be a reasonable way to setup the broker side of the communication, it does not serve external tools (CLI) like the Zookeeper Migration Tool. As well has been considered to reject this option, that this will mean users need two different places for configuration. As environment variables for TLS communication with zookeeper, and others as properties files like the zookeeeper.connect and others.

Give security configuration as parameters for CLI tools

Every config can be prefixed with "authorizer." for the case when kafka.security.authorizer.AclAuthorizer requires a different TLS configuration when connecting to ZooKeeper.  Each can config can be individually prefixed at will, and any prefixed config will override the corresponding Kafka config value.

As noted above, ZooKeeper does not support setting a key password within the keystore that differs from the keystore password itself: trying to do so will cause the ZooKeeper connection attempt to fail.  Therefore there is no configuration value for the key password, and the key password must be identical to the keystore password.

ZooKeeper Security Migration CLI

The existing configuration option java.security.auth.login.config, used to set provide the JAAS configuration content will be kept as it is. 

A new parameter will be added:

• --zk-tls-config-file: <String: Zookeeper TLS configuration file path>

This config file may contain any of the above properties to configure the TLS connection to ZooKeeper; any config(s) not mentioned in the above list will be ignored (a convenience to make it possible to pass a Broker properties file)

Config Command CLI 

The same --zk-tls-config-file parameter will be added to support the bootstrap use case.

ACL Command CLI 

The same --zk-tls-config-file parameter will be added to support the bootstrap use case.

ZooKeeper Shell CLI 

A -zk-tls-config-file parameter will be added.  Note the use of single-dash as opposed to double-dash here since all of the tool's parameters follow the ZooKeeper project's style and are specified via a single dash.  The "usage" message will be updated accordingly.

Proposed Changes

The proposed changes include the public interface changes:

• New Kafka configurations, both non-prefixed as well as prefixed with "authorizer."
• A new --zk-tls-config-file parameter for:
  • ZooKeeper Security Migration Tool
  • Config Command CLI (for the special use case of bootstrapping TLS-enabled ZooKeeper)
  • ACL Command CLIs (for the special use case of bootstrapping TLS-enabled ZooKeeper)
• A new -zk-tls-config-file parameter in the ZooKeeper Shell (again, note the single dash as opposed to the double-dash used above)

The proposed changes also include the addition of:

• System tests to confirm the hardened/secured configuration for TLS connectivity to ZooKeeper
• The use of ZooKeeper Security Migrator and Kafka Brokers with client certificate authentication both with and without SASL
• Explicit Kafka documentation on how to configure TLS connectivity to ZooKeeper – both mutual TLS and encryption-only

Compatibility, Deprecation, and Migration Plan

The changes are additions only, and there is no compatibility issue in the broker because the default for the broker config zookeeper.client.secure is false; TLS to ZooKeeper is an opt-in.

Test Plan

System tests will cover the following:

• Migrating Zookeeper/Kafka clusters from non-TLS-enabled ZooKeeper to TLS-enabled ZooKeeper
• Invoking the Zookeeper Security Migration tool against TLS-enabled ZooKeeper both with and without ZK SASL authentication enabled
• TLS encryption-only (i.e. no client certificate) connectivity to ZooKeeper.

Compatibility testing is unnecessary because Zookeeper TLS is not available in prior versions.

The connection between Kafka and Zookeeper is not on a critical path related to performance – brokers don't repeatedly communicate with Zookeeper as they process messages, for example – so introducing TLS encryption here does not require explicit performance testing.

Rejected Alternatives

ZooKeeper-Style Configuration Names

ZooKeeper uses camelCase configs that are inconsistent with Kafka broker configs:

It would be confusing and prone to mistake to have such a mismatch – especially for people who tend to know very little about ZooKeeper compared to Kafka.

The ZooKeeper config "zookeeper.client.secure" is also confusing – it only refers to whether the client should be configured to communicate with a TLS-encrypted ZK socket and has nothing to do with any other security-related client configuration that might be implied by the term "secure" (e.g. SASL, ACLs).  We therefore use the broker-side config "zookeeper.ssl.client.enable" instead.

Inheriting Broker Configuration Values

As mentioned above, certain broker configuration related to TLS cannot be inherited because keystore and trustore information is dynamically reconfigurable and may end up being stored in ZooKeeper.  There are other TLS configuration values that are not dynamically reconfigurable in the broker (protocols and cipher suites, for example), but selectively inheriting these configs provides little value and could simply introduce confusion as people might assume – incorrectly – that keystore and truststore information could also be inherited.  We therefore inherit nothing from the broker related to TLS configuration.

Requiring All Authorizer Prefixed TLS Configs

We considered requiring all authorizer ZooKeeper TLS configuration values (prefixed with "authorizer.") to be specified if any of them were specified with a prefix.  This seemed onerous and unnecessary, and it was inconsistent with the way other ZooKeeper configs for the authorizer worked:  they are overlays/overrides, the Kafka config is inherited if they aren't specified via the prefix, and any/all can be prefixed or left out.  So we treat the ZooKeeper TLS configs the same way.

zookeeper.ssl.hostname.verifiction.enable

We could opt to use a true/false config to enable/disable ZooKeeper hostame verification.  The ZooKeeper system property zookeeper.ssl.hostnameVerification works that way (and cannot be changed).  However, Kafka uses a different convention: it clears the endpoint identification algorithm from its default value of https to disable hostname verification.  Since we are explicitly deviating from the ZooKeeper system properties everywhere else, and since this config is rarely used, we will stay consistent with the Kafka config here as well.

zookeeper.ssl.context.supplier.class

The "zookeeper.ssl.context.supplier.class" configuration doesn't actually exist in ZooKeeper 3.5.7.  The ZooKeeper admin guide documents it as being there, but it doesn't appear in the code.  This means we can't support it in this KIP.  It has been added in the latest ZooKeeper 3.6 SNAPSHOT, so this config could probably be added to Kafka via a new, small KIP if/when we upgrade to ZooKeeper 3.6 (which looks to be in release-candidate stage as of this writing).