...

Since the merge KAFKA-8634 (https://github.com/apache/kafka/commit/d67495d6a7f4c5f7e8736a25d6a11a1c1bef8d87)   in trunk, Apache Kafka can work ships with the latest Apache Zookeeper supporting TLS and Dynamic Reconfiguration (AK 2.4 ultimately shipped with ZooKeeper version 3.5.6 rather than 3.5. 

This version comes with two very expected features, the support for TLS connections and the introduction of Dynamic Reconfiguration.

The current internal zookeeper client version, at the time of this KIP, support JAAS, SASL, based communication credentials. When 5, but the general functionality is the same).  When doing a deployment in a security-minded environment the user aim desire is to use TLS to secure encryption in transitencrypt communication in transit.

It is possible to enable TLS connectivity to Zookeeper from Apache Kafka 2.4 -- the problem is that configuration information has to be passed via system properties as -D command line options on the Java invocation of the broker or CLI tool (e.g. ZooKeeper Security Migration), and such -D  command line options are not secure because anyone with access to the box can see the command line used to start the process; the configuration includes sensitive keystore/truststore password information, so we need a secure mechanism for passing the configuration values. The motivation for this KIP is to harden/secure the configuration mechanism for Zookeeper TLS connectivity.

With this KIP we aim to introduce the necessary changes to enable the possibility to use a use of secure configuration values when defining TLS encrypted channel channels for communications with Zookeeper. These changes will enable the secure use of TLS with the Zookeeper Security Migration CLI as well within the internal usages.

Goals

• Extend the internal zookeeper client to get external configuration from a file.
• Extend the Zookeeper Security Migration CLI to allow passing a property.
• Add support for the necessary zookeeper SSL configuration variables within the server.properties file.

Public Interfaces

Zookeeper Security Migration CLI 

The existing configuration option java.security.auth.login.config, used to set provide the JAAS configuration content will be kept as it is. 

A new parameter will be added:

• zookeeper.config.path: This CLI parameter will allow users to pass a zookeeper properties file with the relevant configuration to be used to connect with the zookeeper server.

The zookeeper.config.path parameter will take precedence over the java.security.auth.login.config option.

Config Command CLI 

Still some other CLIs have the option have the option to connect to ZK. This commands still support existing java.security.auth.login.config to support a SASL/JAAS based authentication. For the purpose of this KIP a new parameter will be added:

• zookeeper.config.path: This CLI parameter will allow users to pass a zookeeper properties file with the relevant configuration to be used to connect with the zookeeper server.

The zookeeper.config.path parameter will take precedence over the java.security.auth.login.config option.

Other command that will undergo the same transformation are:

• PreferredReplicaLeaderElectionCommand
• TopicCommand
• ReassignPartitionsCommand

Kafka Configuration file (server.properties)

from brokers as well as any CLI tools that will still contain non-deprecated direct ZooKeeper communication in the next AK (AK 2.5) release.

The list of CLI tools that used non-deprecated direct ZooKeeper access in the previous AK (AK 2.4) release was as follows:

1. zookeeper-security-migration.sh (kafka.admin.ZSecurityMigrator)
2. kafka-reassign-partitions.{bat,sh} (kafka.admin.ReassignPartitionsCommand)
3. kafka-configs.{bat,sh} (kafka.admin.ConfigCommand)

Direct ZK access in #2 above is being addressed via the already-accepted KIP-455: Create an Administrative API for Replica Reassignment.

There is not yet a KIP to address direct ZK access in #3, and in fact ConfigCommand presents a bit of a conundrum because it explicitly states in a comment that a supported use case is bootstrapping a Kafka cluster with encrypted passwords in Zookeeper (see https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/ConfigCommand.scala#L65).  This is a very special use case for sure, but it does mean that it will be especially difficult to fully/100% deprecate this particular direct Zookeeper connectivity without a different storage mechanism for dynamic configuration values being available (e.g. the self-managed quorum referred to in KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum).

It doesn't make sense to address direct ZK access in the ZkSecurityMigrator since connecting to ZooKeeper and applying/removing ACLs is the whole point of the tool.  (In theory we could replace its direct ZK access in favor of a Kafka API, but that seems silly.)

ZooKeeper also supports TLS connectivity between ZK nodes for Quorum-related communication.  This is configured independent of Kafka within ZooKeeper.

The class kafka.security.authorizer.AclAuthorizer talks directly to ZooKeeper, so it must be possible to configure that ZooKeeper connection for TLS as well.  Note that kafka.security.auth.SimpleAclAuthorizer was deprecated in AK 2.4 (in favor of AclAuthorizer) and will not support TLS connectivity to ZooKeeper.

Goals

• Harden/secure the configuration mechanism for Zookeeper TLS connectivity from:
  • Kafka Brokers (including from kafka.security.authorizer.AclAuthorizer if/when configured)
  • zookeeper-security-migration.sh
  • kafka-reassign-partitions.{bat,sh} and/or kafka-configs.{bat,sh} assuming direct ZooKeeper connectivity is not deprecated in the next AK release (and if so, then great – no need to do anything)
    
• Add system tests to confirm the hardened/secured configuration for TLS connectivity to ZooKeeper
• Add explicit Kafka documentation on how to configure TLS connectivity to ZooKeeper
• Add a reference in the Kafka documentation to the ZooKeeper Quorum TLS configuration (https://zookeeper.apache.org/doc/r3.5.6/zookeeperAdmin.html#Communication+using+the+Netty+framework)

Out of Scope

• Zookeeper-to-Zookeeper Quorum TLS system tests and in-depth documentation (the ZooKeeper project already has such tests and documentation)
• Kafka API for kafka.admin.ConfigCommand and deprecating its direct ZooKeeper connectivity (this must be addressed as a separate KIP)

Public Interfaces

New Broker and AclAuthorizer Configurations

The below table contains the complete list of added configs.  All configs being added are optional Strings with no default value unless otherwise noted.  As an example, these are some of the configs that will be introduced:

zookeeper.client.secure=trueNew configuration properties will be added to the KafkaConfig object. As an example, these are the ones that will be introduced:
zookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
zookeeper.

...

ssl.keyStore.location=kafka.jks
zookeeper.ssl.keyStore.password=test1234
zookeeper.ssl.trustStore.location=truststore.jks
zookeeper.ssl.trustStore.password=test1234

Simple ACL Authorizer configuration (server.properties)

Currently the Simple ACL authorizer had the option to use different zookeeper connection settings. With this KPI we will need to add the variables need to connect with TLS. 

With this KPI we will add this variables:

...

Every config can be prefixed with "authorizer." for the case when kafka.security.authorizer.AclAuthorizer connects via TLS to a ZooKeeper quorum separate from the one that Kafka is using – this specific use case will be identified in the configuration by explicitly setting authorizer.zookeeper.client.secure=true.  In this case the configs prefixed with "authorizer." are not "overrides" like the other authorizer ZooKeeper connectivity configs such as connection/session timeouts and max inflight requests; ZooKeeper TLS connectivity values for the authorizer are not "merged" with Kafka's ZooKeeper TLS configs (if any) because semantically the two sets of configs are for different ZooKeeper quorums and there is no guarantee that they would be applicable across the two quorums; any configs that need to be identical across the two ZooKeeper quorums will have to be repeated with and without the prefix.  The same defaults described below will apply to the prefixed configs.

...

Proposed Changes

CLI tools

Each CLI tool that still allow to use zookeeper like:

• PreferredReplicaLeaderElectionCommand
• TopicCommand
• ReassignPartitionsCommand
• ConfigCommand
• Zookeeper Security Migration

Will be adapted to use a config file as a compliment to the current JAAS file method.

KafkaConfiguration

Several new variables, required to have a TLS connection with zookeeper will be added to the configuration definition. These variables will be passed across and used internally when necessary during the creation of the KafkaZkClient objects.

KafkaZkClient

Extend the apply method to hold a Map[String, Any] to support passing any number of configuration parameters being passed to the low level Zk client.

Compatibility, Deprecation, and Migration Plan

The changes are planned to be introduced in a compatible way, by keeping the current JAAS variable precedence. Current users will not see any change, but if a user would like to use a zookeeper server with a TLS connection, it will be able to pass a config file, or update the server.properties file.

Test Plan

A new integration test will be adapted to use the new configurations for TLS connection with Zookeeper. If need relevant existing integration and system tests will be updated accordingly.

Rejected Alternatives

Use the global system variables as proposed in the Zookeeper User guide

The Zookeeper user guide (https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZooKeeper+SSL+User+Guide) proposes the usage of two environment variables, CLIENT_JVMFLAGS and SERVER_JVMFLAGS, to configure the usage of TLS in the client and server zookeeper sides.

This option has been considered and rejected, while it could be a reasonable way to setup the broker side of the communication, it does not serve external tools (CLI) like the Zookeeper Migration Tool. As well has been considered to reject this option, that this will mean users need two different places for configuration. As environment variables for TLS communication with zookeeper, and others as properties files like the zookeeeper.connect and others.

Give security configuration as parameters for CLI tools

ZooKeeper Security Migration CLI

The existing configuration option java.security.auth.login.config, used to set provide the JAAS configuration content will be kept as it is. 

A new parameter will be added:

• --zk-tls-config-file: <String: Zookeeper TLS configuration file path>

This config file may contain any of the above properties to configure the TLS connection to ZooKeeper; any config(s) not in the above list will be ignored (a convenience to make it possible to pass a Broker properties file)

Config Command CLI 

The same --zk-tls-config-file parameter will be added if (and only if) direct ZooKeeper connectivity for this tool is not deprecated in this Kafka release.

Reassign Partitions CLI 

The same --zk-tls-config-file parameter will be added if (and only if) direct ZooKeeper connectivity for this tool is not deprecated in this Kafka release (i.e. via KIP-455: Create an Administrative API for Replica Reassignment)

Proposed Changes

The proposed changes include the public interface changes:

• New Kafka configurations, both non-prefixed as well as prefixed with "authorizer."
• A new --zk-tls-config-file parameter in the ZooKeeper Security Migration Tool and potentially the Config Command and Reassign Partitions Command CLIs if (and only if) their direct ZooKeeper connectivity is not deprecated in the next release.

The proposed changes also include the addition of:

• system tests to confirm the hardened/secured configuration for TLS connectivity to ZooKeeper
• explicit Kafka documentation on how to configure TLS connectivity to ZooKeeper

Compatibility, Deprecation, and Migration Plan

The changes are additions only, and there is no compatibility issue because the default for zookeeper.client.secure is false.

Test Plan

System tests will cover the following:

• Migrating Zookeeper/Kafka clusters from non-TLS-enabled ZooKeeper to TLS-enabled ZooKeeper
• Invoking the Zookeeper Security Migration tool against TLS-enabled ZooKeeper

Compatibility testing is unnecessary because Zookeeper TLS is not available in prior versions.

The connection between Kafka and Zookeeper is not on a critical path related to performance – brokers don't repeatedly communicate with Zookeeper as they process messages, for example – so introducing TLS encryption here does not require explicit performance testing.

Rejected Alternatives

N/AThis option would make the usage of the CLI tools necessary more complex without adding any more benefits.  As well this will not add any changes in the internal communication.