Status

Current state: Under discussion Accepted

Discussion thread: N/A apache mail archive link

JIRA: KAFKA-2639, KAFKA-2640, KAFKA-2641

Released:  <Kafka Version>0.9.0.0

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

...

•  Apache ZooKeeper documentation
• Apache ZooKeeper wiki
• Cloudera ZooKeeper security configuration

Paths affected

The following paths will have secure ACLs once this feature is turned on:

  val BrokerIdsPath = "/brokers/ids"
  val BrokerTopicsPath = "/brokers/topics"
  val ControllerPath = "/controller"
  val ControllerEpochPath = "/controller_epoch"
  val ReassignPartitionsPath = "/admin/reassign_partitions"
  val DeleteTopicsPath = "/admin/delete_topics"
  val PreferredReplicaLeaderElectionPath = "/admin/preferred_replica_election"
  val BrokerSequenceIdPath = "/brokers/seqid"
  val IsrChangeNotificationPath = "/isr_change_notification"
  val EntityConfigPath = "/config"
  val EntityConfigChangesPath = "/config/changes"

Command-line tools

Command line tools need to have the system property java.security.auth.login.config set pointing to a file with the appropriate credentials so that it has write access to znodes.

Proposed Changes

There are three parts to this new feature:

...

Compatibility, Deprecation, and Migration Plan

Unsecure cluster -> Secure cluster

If a broker all of a sudden changes the ACLs of znodes to secure, then brokers without the proper configuration will not be able to access ZooKeeper data any longer. Consequently, we need a plan to migrate existing clusters without requiring such a cluster to stop to reconfigure. We currently propose the following recipe:

...

ZooKeeper also allows users to disable authentication on the client side even in the presence of a JAAS login file with the property zookeeper.sasl.client. Setting it to false disables client authentication. Additionally, the context key in the JAAS login file is "Client" by default, but that name can be changed by using setting the property zookeeper.sasl.clientconfig.

Secure cluster -> Unsecure cluster

It is desirable to have the ability of bringing a cluster from secure to unsecure, making all znodes open to reads and writes by any client able to connect to the ZooKeeper ensemble. This option might be needed for example when there are issues with security features and they need to be off to allow the cluster to keep operating.

The path to execute this change is the following:

1. Perform a rolling restart setting the JAAS login file, which enables brokers to authenticate, but setting zookeeper.set.acl to false. At the end of the rolling restart, brokers stop creating znodes with secure ACLs, but are still able to authenticate and manipulate all znodes.
2. Execute the ZkSecurityMigrator (there is a script under ./bin and the code is under kafka.admin). This tool traverses the corresponding sub-trees changing the ACLs of the znodes.
3. Perform a second rolling restart of brokers, this time omitting the system property that sets the JAAS login file.

Step 3 isn't strictly necessary, but it is best to just turn off authentication.

Shared ensemble

The changes should not prevent multiple applications sharing an ensemble. One typical way of sharing a ZooKeeper ensemble is to use the chroot feature to create different sub-trees for applications.

The currentl implementation sets an ACL to the root of the Kafka cluster sub-tree, which restricts operations on znodes like deleting children. If chroot is not used, then the ability of sharing the ensemble is limited and will possibly require manual intervention.

Credentials

It should enable different credentials for admin tools and brokers. Sharing credentials between brokers and tools might not be desirable from an operational perspective and we need to enable different credentials to manipulate the appropriate ZooKeeper state (znodes). This aspect isn't entirely clear and is one feature that is not implemented yet.

Rejected Alternatives

One way to restrict access to a ZooKeeper ensemble is to use firewalls. This approach is reasonable, but difficult to implement in a fine-grained manner, which ends up leaving the metadata exposed all the same. Using traffic filtering to complement the feature described here is certainly a recommended option, so this option is not really rejected, but it is deemed insufficient.