You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 2 Next »

Status

Current state: "Under Discussion"

Discussion thread: here

JIRA: Unable to render Jira issues macro, execution error.

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Since the merge KAFKA-8634 (https://github.com/apache/kafka/commit/d67495d6a7f4c5f7e8736a25d6a11a1c1bef8d87)  in trunk, Apache Kafka can work with the latest Apache Zookeeper 3.5.5. 

This version comes with two very expected features, the support for TLS connections and the introduction of Dynamic Reconfiguration.

The current internal zookeeper client version, at the time of this KIP, support JAAS, SASL, based communication credentials. When doing a deployment in a security minded environment the user aim to use TLS to secure encryption in transit.

With this KIP we aim to introduce the necessary changes to enable the possibility to use a TLS encrypted channel for communications with Zookeeper. These changes will enable the use of TLS with the Zookeeper Security Migration CLI as well within the internal usages.

Goals

  • Extend the internal zookeeper client to get external configuration from a file.
  • Extend the Zookeeper Security Migration CLI to allow passing a property.
  • Add support for the necessary zookeeper SSL configuration variables within the server.properties file.

Public Interfaces

Zookeeper Security Migration CLI 

The existing configuration option java.security.auth.login.config, used to set provide the JAAS configuration content will be kept as it is. 

A new parameter will be added:


  • zookeeper.config.path: This CLI parameter will allow users to pass a zookeeper properties file with the relevant configuration to be used to connect with the zookeeper server.


The zookeeper.config.path parameter will take precedence over the java.security.auth.login.config option.

Config Command CLI 

Still some other CLIs have the option have the option to connect to ZK. This commands still support existing java.security.auth.login.config to support a SASL/JAAS based authentication. For the purpose of this KIP a new parameter will be added:


  • zookeeper.config.path: This CLI parameter will allow users to pass a zookeeper properties file with the relevant configuration to be used to connect with the zookeeper server.


The zookeeper.config.path parameter will take precedence over the java.security.auth.login.config option.

Other command that will undergo the same transformation are:

  • PreferredReplicaLeaderElectionCommand
  • TopicCommand
  • ReassignPartitionsCommand

Kafka Configuration file (server.properties)

New configuration properties will be added to the KafkaConfig object. As an example, these are the ones that will be introduced:

zookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
zookeeper.client.secure=true
zookeeper.ssl.keyStore.location=kafka.jks
zookeeper.ssl.keyStore.password=test1234
zookeeper.ssl.trustStore.location=truststore.jks
zookeeper.ssl.trustStore.password=test1234

Simple ACL Authorizer configuration (server.properties)

Currently the Simple ACL authorizer had the option to use different zookeeper connection settings. With this KPI we will need to add the variables need to connect with TLS. 

With this KPI we will add this variables:

authorizer.zookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
authorizer.zookeeper.client.secure=true
authorizer.zookeeper.ssl.keyStore.location=kafka.jks
authorizer.zookeeper.ssl.keyStore.password=test1234
authorizer.zookeeper.ssl.trustStore.location=truststore.jks
authorizer.zookeeper.ssl.trustStore.password=test1234

Proposed Changes

CLI tools

Each CLI tool that still allow to use zookeeper like:

  • PreferredReplicaLeaderElectionCommand
  • TopicCommand
  • ReassignPartitionsCommand
  • ConfigCommand
  • Zookeeper Security Migration

Will be adapted to use a config file as a compliment to the current JAAS file method.

KafkaConfiguration

Several new variables, required to have a TLS connection with zookeeper will be added to the configuration definition. These variables will be passed across and used internally when necessary during the creation of the KafkaZkClient objects.

KafkaZkClient

Extend the apply method to hold a Map[String, Any] to support passing any number of configuration parameters being passed to the low level Zk client.

Compatibility, Deprecation, and Migration Plan

The changes are planned to be introduced in a compatible way, by keeping the current JAAS variable precedence. Current users will not see any change, but if a user would like to use a zookeeper server with a TLS connection, it will be able to pass a config file, or update the server.properties file.

Test Plan

A new integration test will be adapted to use the new configurations for TLS connection with Zookeeper. If need relevant existing integration and system tests will be updated accordingly.

Rejected Alternatives

Use the global system variables as proposed in the Zookeeper User guide

The Zookeeper user guide (https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZooKeeper+SSL+User+Guide) proposes the usage of two environment variables, CLIENT_JVMFLAGS and SERVER_JVMFLAGS, to configure the usage of TLS in the client and server zookeeper sides.

This option has been considered and rejected, while it could be a reasonable way to setup the broker side of the communication, it does not serve external tools (CLI) like the Zookeeper Migration Tool. As well has been considered to reject this option, that this will mean users need two different places for configuration. As environment variables for TLS communication with zookeeper, and others as properties files like the zookeeeper.connect and others.

Give security configuration as parameters for CLI tools

This option would make the usage of the CLI tools necessary more complex without adding any more benefits.  As well this will not add any changes in the internal communication.

  • No labels