Server-Server mutual authentication

server.id=FQDN:port:port

uring leader activation(leader-election) phase, all the servers in an ensemble will participate to form quorum. By default this communication is not authenticated. This guide describes how to enable secure communication between the quorum peer servers using SASL mechanism. ZooKeeper supports Kerberos or DIGEST-MD5 as your authentication scheme.

Presently, this feature is supported only in ZooKeeper 3.4.10+ version and is implemented using JIRA ZOOKEEPER-1045. The feature code will be forward ported to ZooKeeper 3.5.x and 3.6.x versions in a separate JIRA task ZOOKEEPER-2639.

ZooKeeper SASL configurations:

Following configurations should be added in $conf/zoo.cfg configuration file.

quorum.auth.enableSasl=true
quorum.auth.learnerRequireSasl=true
quorum.auth.serverRequireSasl=true
quorum.auth.learner.loginContext=QuorumLearner
quorum.auth.server.loginContext=QuorumServer
quorum.auth.kerberos.servicePrincipal=servicename/_HOST
quorum.cnxn.threads.size=20 

2.1) Sets to enable quorum authentication using SASL.

# Defaulting to false
quorum.auth.enableSasl=true

2.2) Sets to connect using quorum authentication. If this is true, quorum peer learner will send authentication packet to quorum peer server then proceeds with LE on successful authentication. If false, then proceeds with LE without any authentication. This can be used while upgrading ZooKeeper server.

# Defaulting to false
quorum.auth.learnerRequireSasl=true

2.3) Sets to connect using quorum authentication. If this is true, then all unauthenticated quorum peer learner connection requests will be rejected. If false, then quorum peer server will accept quorum peer learner connection request and then proceeds with Leader Election even if the authentication did not succeed. This can be used while upgrading ZK server.

# Defaulting to false
quorum.auth.serverRequireSasl=true

2.3)  (Optional) If you want to use different login context for learner/server other than the default values, then configure the following:

# Defaulting to QuorumLearner
quorum.auth.learner.loginContext=QuorumLearner

# Defaulting to QuorumServer
quorum.auth.server.loginContext=QuorumServer

2.4)  The maximum number of threads to allow in the “connectionExecutors” thread pool, which will be used to process quorum server connection requests during Leader Election. This needs to be tuned depends on the cluster size.

# Defaulting to 20quorum.cnxn.threads.size=20

Add the following settings to the java.env file located in the ZooKeeper configuration directory. (Create the file if it does not already exist.)

export JVMFLAGS="-Djava.security.auth.login.config=/etc/zk/conf/jaas.conf"

 ZooKeeper servers will talk each other using the credentials configured in “jaas.conf” file. They will act like client-server when creating connections during quorum formation. Set up the Java Authentication and Authorization Service (JAAS) by creating a “jaas.conf” file in the ZooKeeper configuration directory and add configuration entries into this file specific to your selected authentication schemes.

Following section describes the details of supported authentication schemes, Kerberos or DIGEST-MD5.

Kerberos based authentication:

ZooKeeper uses Kerberos principals and Keytabs to support quorum peer mutual authentication. Kerberos assigns tickets to Kerberos principals to enable them to do the Kerberos-secured communication.

You can use principal in the form of servicename/fully.qualified.domain.name@EXAMPLE.COM, here "servicename" has to be substituted with the proper service name. For example, for ZooKeeper quorum service you can substitute "servicename" with "zkquorum". Assume, we have two hosts (myhost1.foo.com and myhost2.foo.com) we would have Kerberos principals like, zkquorum/myhost1.foo.com@EXAMPLE.COM and zkquorum/myhost2.foo.com@EXAMPLE.COM

Configure service name and host details servicename/fully.qualified.domain.name of quorum peer server to the following configuration property in zoo.cfg file. This service principal will be used by the quorum learner to send authentication packet to the peer quorum server.

The principal name should be in either of the following formats:

1)  Single shared Kerberos principal name configured in all the servers.

servicename/localhost@EXAMPLE.COM

Admin has to be configure the below property with the principal name in zoo.cfg file,

quorum.auth.kerberos.servicePrincipal=servicename/localhost

 Important Note: Authorization is not supported in this format.

2)  Host based Kerberos principal name with _HOST wildcard.

servicename/fully.qualified.domain.name@EXAMPLE.COM

ZooKeeper simplifies the deployment of configuration files by allowing the fully qualified domain name component of the service principal to be specified as the _HOST wildcard. Internally each quorum learner will substitute _HOST with the respective FQDN from zoo.cfg at runtime and then send authentication packet to that server. This allows administrators to avoid the overhead of configuring all others principal names on all nodes. However, the keytab files will be different. A keytab file for a ZooKeeper server is unique to each host if the principal name contains the hostname. This file is used to authenticate a principal on a host to Kerberos without human interaction or storing a password in a plain text file. Access to the keytab files should be tightly secured because having access to the keytab file for a principal allows one to act as that principal.

ZooKeeper ensemble server details will be configured in zoo.cfg of the form

server.id=host:port:port

Make sure that you should use FQDN in zoo.cfg like below, FQDN value will be used to replace the special string pattern “_HOST”

server.id=FQDN:port:port