During leader activation(leader-election) phase, all the servers in an ensemble will participate to form quorum. By default, this communication is not authenticated. This guide describes how to enable secure communication between the quorum peer servers using SASL mechanism. ZooKeeper supports Kerberos or DIGEST-MD5 as your authentication scheme.

...

2.4)  The maximum number of threads to allow in the “connectionExecutors” thread pool, which will be used to process quorum server connection requests during Leader Election. This needs has to be tuned depends depending on the cluster size.

# Defaulting to 20quorum.cnxn.threads.size=20

...

 ZooKeeper servers will talk to each other using the credentials configured in “jaas.conf” file. They will act like clientlearner-server when creating connections during quorum formation. Set up the Java Authentication and Authorization Service (JAAS) by creating a “jaas.conf” file in the ZooKeeper configuration directory and add configuration entries into this file specific to your selected authentication schemes.

...

ZooKeeper simplifies the deployment of configuration files by allowing the fully qualified domain name component of the service principal to be specified as the _HOST wildcard. Internally each quorum learner will substitute _HOST with the respective FQDN from zoo.cfg at runtime and then send authentication packet to that server. This allows administrators to avoid the overhead of configuring all others other's principal names on all nodes. However, the keytab files will be different. A keytab file for a ZooKeeper server is unique to each host if the principal name contains the hostname. This file is used to authenticate a principal on a host to Kerberos without human interaction or storing a password in a plain text file. Access to the keytab files should be tightly secured because having access to the keytab file for a principal allows one to act as that principal.

...

server.id=host:port:port

Make sure that you should use FQDN in zoo.cfg like below, FQDN value will be used to replace the special string pattern “_HOST”

...

ZooKeeper server will support authorization if the principal name is in the format servicename/fully.qualified.domain.name@EXAMPLE.COM and configure “_HOST” wildcard in quorum server principal in zoo.cfg file likeas below,

quorum.auth.kerberos.servicePrincipal= servicename/_HOST

Now, QuorumServer will do the authorization checks against configured authorized hosts. This authorized host list will be prepared using the ensemble server details in zoo.cfg file. During LE, QuorumLearner will send an authentication packet to QuorumServer. Now, QuorumServer will check that the connecting QuorumLearner’s hostname should exists in the authorized hosts. If not exists then connecting peer is not authorized to join this ensemble and the request will be rejected immediately.

For example, zoo.cfg contains the below server details. Now, only FQDN1, FQDN2, FQDN3 hosts are allowed to join ZooKeeper cluster. All others will be rejected as unauthorized connections.

server.1=FQDN1:port:port
server.2=FQDN2:port:port
server.3=FQDN3:port:port

...

Rolling upgrade:

This feature is targeted for supported in 3.4 branch which is the current stable branch and is widely used in production clusters. So it is critical to support rolling upgrade of existing cluster without any down time. This section describing describes the possible rolling upgrade.

...

quorum.auth.learnerRequireSasl: Initially false. Sets to true in second step of rolling upgrade, this is to prepare each server ready to send authentication packet to other servers. Can’t This flag can’t be set to false if quorum.auth.serverRequireSasl serverRequireSasl is set to true.

quorum.auth.serverRequireSasl: Initially false. Sets to true in third step of rolling upgrade (quorum.auth.learnerRequireSasl should be true) to enable server-to-­server SASL authentication strictly.

...

Step-3) Loop through each server, update zoo.cfg with quorum.auth.serverRequireSasl=true, then restart server. Now for each server, both its sending part (modeled as QuorumLearner) and its receiving part (modeled as QuorumServer) are auth enabled. Verify everything works correctly after restarting the server.

For verification, you can use ZooKeeper client operations like, create/delete znode.

...