Security

The following is a proposal for securing Kafka. We don't know this area that well, so this document is as much notes and exploration of the space as a spec.

Requirements

• Need to support both Kerberos and TLS (SSL)
• Need to support unix-like users and permissions
• Need some kind of group or ACL notion
• No backwards-incompatible release

Assumptions

• On-disk encryption is not needed
• We can secure Zookeeper
• Not trying to run Kafka on the open internet

Authentication

We are tentatively planning to use SASL.

Different authentication mechanisms will be available on different ports. We will continue to support the existing unauthenticated communication, but add a new secure.port for SASL-based authentication.

All connections on the unauthenticated port will be assigned a fake user ("nobody" or "josephk" or something).

Implementing Authentication

The acceptor thread that currently handles socket acceptance will become responsible for authenticating the user as themselves. This thread will add the user Subject or some proxy for this to the socket's context information. This subject will be passed along by the socket server as part of each Request object sent on the RequestChannel. Authorization will be handled in the API layer.

Authorization

The plan will be to support unix-like permissions on a per-topic level.

An important question is whether to support traditional groups or more flexible acls. I don't understand this yet.

We will have the following permissions:

READ - Permission to fetch data from the topic
WRITE - Permission to publish data to the topic
DELETE - Permission to delete the topic
CREATE - Permission to create the topic
CONFIGURE - Permission to change the configuration for the topic
LIST - Permission to fetch metadata on the topic
REPLICATE - Permission to participate as a replica (i.e. issue a fetch request with a non-negative node id)

Permission are not hierarchical since topics are not hierarchical. So a user will have a default value for these (a kind of umask) as well as a potential override on a per-topic basis. Note that CREATE permission primarily only makes sense at the default level.

We will maintain permissions for each topic in a manner similar to the handling of configs. We will have a zookeeper directory
/permissions/defaults
which contains the default permissions as well as
/permissions/topics
which will have per-topic permission settings.

For each secured action on the server, the server will do a permissions check like
permissionsManager.ensurePermitted(request.user, Permissions.Read, "mytopicname")

Obviously this check will have to be fast since it will be done at least once on most requests, so the necessary permissions will need to be cached locally.

Open Questions

• On-the-wire encryption: do we need to do this? If so we will have to disable the sendfile optimization when encryption is used.
• Groups vs ACLs: need to understand pros and cons.
• Can we do everything over a single port?

Notes