You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 2 Next »

Status

Current state: Under discussion

Discussion thread: N/A

JIRA: KAFKA-2639, KAFKA-2640, KAFKA-2641

Released: <Kafka Version>

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

As part of the effort to introduce and strengthen the security features of Kafka, we propose a set of changes to authenticate access to the metadata stored in ZooKeeper. Currently, the metadata stored in ZooKeeper for any given Kafka cluster is open and can be manipulated by any client with access to the ZooKeeper ensemble. The goal of this KIP is to restrict access to authenticated clients by leveraging the SASL authentication feature available in the 3.4 branch of ZooKeeper. With this feature, clients authenticate upon connecting to a ZooKeeper server with security configured and enabled. Once authenticated, the ensemble uses the client credentials to authorize access to znodes that have ACLs configured to restrict access.

The current ACL used with all znodes is open for read and write access, and with this KIP we want to make it unrestricted for authenticated clients, but readable by any client. We assume that no data stored in ZooKeeper is sensitive and can be read by anyone with the ability of connecting to the ensemble, but the metadata stored in it can be used to mount specific attacks against the cluster. For example, the metadata of a broker can be manipulated or a rogue broker can be introduced and start participating in replica sets as any other broker. Note that the feature described here focuses on brokers and does not include the old consumer, since it is becoming deprecated.

Public Interfaces

Users need to setup a JAAS login configuration file and specify it as a system property named {{java.security.auth.login.config}}. With this property set, Kafka brokers turn security features on and use more strict ACLs rather than the open unsafe one. Specifically, it uses CREATOR_ALL_ACL and READ_ACL_UNSAFE when the security feature is on, which enable the creator (or anyone with the credentials of the creator) to manipulate the znode while everyone else can read it. That's the only bit that changes to users, everything else happens under the hood.

ZooKeeper currently offers two mechanisms of authentication: Kerberos and DIGEST-MD5. For more information on the options, check the following online docs:

Proposed Changes

There are three parts to this new feature:

  • ZkUtils: Pretty much all access to ZooKeeper happens through kafka.utils.ZkUtils and the JIRA for these changes is KAFKA-2639. The main problem with the code before the changes in KAFKA-2639 is the fact that ZkUtils is a singleton object, which makes changing the ACLs from not secure to secure difficult. We need to be able to change ACLs dynamically at least for testing, however. In KAFKA-2639, we make ZkUtils a class and now we can determine if we want to use strict or open ACLs at instantiation.
  • Configuration and upgrades: The user needs to be able to input configuration to express the interest in running a secure cluster and needs an upgrade path to bring non-secure clusters to secure. The main part of the configuration is in the JAAS login file, and we pass it to a broker via a system property. The upgrade path is discussed below, and this part of the proposal is in KAFKA-2641.
  • Tests: This task is mainly about changing existing test cases and creating new ones to test new functionality. This part of the work is in KAFKA-2640. 

Compatibility, Deprecation, and Migration Plan

If a broker all of a sudden changes the ACLs of znodes to secure, then brokers without the proper configuration will not be able to access ZooKeeper data any longer. Consequently, we need a plan to migrate existing clusters without requiring such a cluster to stop to reconfigure. We currently propose the following recipe:

  1. Perform a rolling restart setting the JAAS login file, which enables brokers to authenticate. At the end of the rolling restart, brokers are able to manipulate znodes with strict ACLs, but they will not create znodes with those ACLs.
  2. Execute a tool called ZkSecurityMigrator (thre is a script under ./bin and . This tool traverses the corresponding sub-trees changing the ACLs of the znodes.
  3. Perform a second rolling restart of brokers, this time setting the configuration parameter zookeeper.set.acl to true, which enables ZkUtils to use secure ACLs when creating znodes.

There is a window of vulnerability in this scheme, however. If a znode is created concurrently with the traversal of the sub-trees by the migration tool, then that znode might end up with an open and unsafe ACL.

Rejected Alternatives

One way to restrict access to a ZooKeeper ensemble is to use firewalls. This approach is reasonable, but difficult to implement in a fine-grained manner, which ends up leaving the metadata exposed all the same. Using traffic filtering to complement the feature described here is certainly a recommended option, so this option is not really rejected, but it is deemed insufficient.

  • No labels