...

•  Apache ZooKeeper documentation
• Apache ZooKeeper wiki
• Cloudera ZooKeeper security configuration

Paths affected

The following paths will have secure ACLs once this feature is turned on:

  val BrokerIdsPath = "/brokers/ids"
  val BrokerTopicsPath = "/brokers/topics"
  val ControllerPath = "/controller"
  val ControllerEpochPath = "/controller_epoch"
  val ReassignPartitionsPath = "/admin/reassign_partitions"
  val DeleteTopicsPath = "/admin/delete_topics"
  val PreferredReplicaLeaderElectionPath = "/admin/preferred_replica_election"
  val BrokerSequenceIdPath = "/brokers/seqid"
  val IsrChangeNotificationPath = "/isr_change_notification"
  val EntityConfigPath = "/config"
  val EntityConfigChangesPath = "/config/changes"

Command-line tools

Command line tools need to have the system property java.security.auth.login.config set pointing to a file with the appropriate credentials so that it has write access to znodes.

Proposed Changes

There are three parts to this new feature:

...

Compatibility, Deprecation, and Migration Plan

Unsecure cluster -> Secure cluster

If a broker all of a sudden changes the ACLs of znodes to secure, then brokers without the proper configuration will not be able to access ZooKeeper data any longer. Consequently, we need a plan to migrate existing clusters without requiring such a cluster to stop to reconfigure. We currently propose the following recipe:

...

ZooKeeper also allows users to disable authentication on the client side even in the presence of a JAAS login file with the property zookeeper.sasl.client. Setting it to false disables client authentication. Additionally, the context key in the JAAS login file is "Client" by default, but that name can be changed by using setting the property zookeeper.sasl.clientconfig.

Secure cluster -> Unsecure cluster

It is desirable to have the ability of bringing a cluster from secure to unsecure, making all znodes open to reads and writes by any client able to connect to the ZooKeeper ensemble. This option might be needed for example when there are issues with security features and they need to be off to allow the cluster to keep operating.

The path to execute this change is the following:

1. Perform a rolling restart setting the JAAS login file, which enables brokers to authenticate, but setting zookeeper.set.acl to false. At the end of the rolling restart, brokers stop creating znodes with secure ACLs, but are still able to authenticate and manipulate all znodes.
2. Perform a second rolling restart of brokers, this time omitting the system property that sets the JAAS login file.
3. Execute the ZkSecurityMigrator (there is a script under ./bin and the code is under kafka.admin). This tool traverses the corresponding sub-trees changing the ACLs of the znodes.

Step 2 isn't strictly necessary, but it is best to just turn off authentication.

Rejected Alternatives

One way to restrict access to a ZooKeeper ensemble is to use firewalls. This approach is reasonable, but difficult to implement in a fine-grained manner, which ends up leaving the metadata exposed all the same. Using traffic filtering to complement the feature described here is certainly a recommended option, so this option is not really rejected, but it is deemed insufficient.