server.id=FQDN:port:port

uring During leader activation(leader-election) phase, all the servers in an ensemble will participate to form quorum. By default this communication is not authenticated. This guide describes how to enable secure communication between the quorum peer servers using SASL mechanism. ZooKeeper supports Kerberos or DIGEST-MD5 as your authentication scheme.

Presently, this feature is supported only in ZooKeeper 3.4.10+ version and is implemented using JIRA ZOOKEEPER-1045. The feature code will be forward ported to ZooKeeper 3.5.x and 3.6.x versions in a separate JIRA task ZOOKEEPER-2639.

ZooKeeper SASL configurations:

Following configurations should be added in $conf/zoo.cfg configuration file.

...

Following section describes the details of supported authentication schemes, Kerberos or DIGEST-MD5.

Kerberos based authentication:

ZooKeeper uses Kerberos principals and Keytabs to support quorum peer mutual authentication. Kerberos assigns tickets to Kerberos principals to enable them to do the Kerberos-secured communication.

...

The principal name should be in either of the following formats:

1)  Single shared Kerberos principal nameconfigured in all the servers.

servicename/localhost@EXAMPLE.COM

Admin has to be configure the below property with the principal name in zoo.cfg file,

quorum.auth.kerberos.servicePrincipal=servicename/localhost

 Important Note: Authorization is not supported in this format.

2)  Host based Kerberos principal name with _HOST wildcard.

servicename/fully.qualified.domain.name@EXAMPLE.COM

...

Make sure that you should use FQDN in zoo.cfg like below, FQDN value will be used to replace the special string pattern “_HOST”HOST”

server.id=FQDN:port:port

QuorumServer {
       com.sun.security.auth.module.Krb5LoginModule required
       useKeyTab=true
       keyTab="/path/to/keytab"
       storeKey=true
       useTicketCache=false
       debug=false
       principal="zkquorum/fully.qualified.domain.name@EXAMPLE.COM";
};

QuorumLearner {
       com.sun.security.auth.module.Krb5LoginModule required
       useKeyTab=true
       keyTab="/path/to/keytab"
       storeKey=true
       useTicketCache=false
       debug=false
       principal="learner/fully.qualified.domain.name@EXAMPLE.COM";
};

Supports authorization:

ZooKeeper server will support authorization if the principal name is in the format servicename/fully.qualified.domain.name@EXAMPLE.COM and configure “_HOST” wildcard in quorum server principal in zoo.cfg file like,

quorum.auth.kerberos.servicePrincipal= servicename/_HOST

Now, QuorumServer will do the authorization checks against configured authorized hosts. This authorized host list will be prepared using the ensemble server details in zoo.cfg file. During LE, QuorumLearner will send an authentication packet to QuorumServer. Now, QuorumServer will check that the connecting QuorumLearner’s hostname should exists in the authorized hosts. If not exists then connecting peer is not authorized to join this ensemble and the request will be rejected immediately.

For example, zoo.cfg contains the below server details. Now, only FQDN1, FQDN2, FQDN3 hosts are allowed to join ZooKeeper cluster. All others will be rejected as unauthorized.

server.1=FQDN1:port:port
server.2=FQDN2:port:port
server.3=FQDN3:port:port

Note:-

It is recommended to set JVM flag (Optional)

javax.security.auth.useSubjectCredsOnly=false

 Reference Links:-

http://docs.oracle.com/javase/7/docs/technotes/guides/security/jgss/tutorials/Troubleshooting.html

http://docs.oracle.com/javase/7/docs/technotes/guides/security/jgss/tutorials/BasicClientServer.html

DIGEST-MD5 based authentication:

QuorumServer {
       org.apache.zookeeper.server.auth.DigestLoginModule required
       user_test="test";
};

QuorumLearner {
       org.apache.zookeeper.server.auth.DigestLoginModule required
       username="test"
       password="test";
};

Important Note: Authorization is not supported in this format

Rolling upgrade:

This feature is targeted for 3.4 branch which is the current stable branch and is widely used in production clusters. So it is critical to support rolling upgrade of existing cluster without any down time. This section describing the possible rolling upgrade.

Rolling upgrade is achieved by a combination of configuration flags in zoo.cfg configuration file. 

quorum.auth.enableSasl: If false, no authentication at all. If true, could be either in rolling upgrade, or finished rolling upgrade.

quorum.auth.learnerRequireSasl: Initially false. Sets to true in second step of rolling upgrade, this is to prepare each server ready to send authentication packet to other servers. Can’t be set to false if quorum.auth.serverRequireSasl set to true.

quorum.auth.serverRequireSasl: Initially false. Sets to true in third step of rolling upgrade (quorum.auth.learnerRequireSasl should be true) to enable server-to-­server SASL authentication strictly.

How to perform rolling upgrade:

Prior to enabling ZooKeeper to work with security on your cluster, make sure ZooKeeper cluster works well in no-authentication setup. Rolling upgrade should be completed in three steps. After every step, admin has to ensure that all the servers have completed this step before moving on to the next step.

Step-1) Loop through each server, update zoo.cfg with quorum.auth.enableSasl=true, then restart the server with the new ZooKeeper 3.4.10+ binaries. Verify everything works well after restarting the server.

Step-2) Loop through each server, update zoo.cfg with quorum.auth.learnerRequireSasl=true, then restart the server. Each server is now ready to talk with other servers in an auth­ enabled way but on the other end the receiving server is not auth enabled yet. Verify everything works well after restarting the server.

Step-3) Loop through each server, update zoo.cfg with quorum.auth.serverRequireSasl=true, then restart server. Now for each server, both its sending part (modeled as QuorumLearner) and its receiving part (modeled as QuorumServer) are auth enabled. Verify everything works correctly after restarting server.

For verification, you can use ZooKeeper client operations like, create/delete znode.