Overview

The following is a proposal for securing Kafka. We don't know this area that well, so this document is as much notes and exploration of the space as a spec.

Requirements

• Need to support both Kerberos and TLS (SSL)
• Need to support unix-like users and permissions
• Need some kind of group or ACL notion
• No backwards-incompatible release
• Encryption at rest supporting compliance regulations
• Non-repudiation and long term non-repudiation for data integrity

Kerberos is the most common for human usage, SSL is more common for applications.

...

• On-disk encryption is not neededneeded 
• We can secure Zookeeper
• Not trying to run Kafka on the open internet

...

Making Kafka safe to run on the open internet requires defending against attacks on availability (DDOS) and other things. This hasn't really been thought through. Our assumption is that the goal is to protect your data against attack but not to defend against availability attacks like creating too many connections or things like that.

Authentication

We are tentatively planning to use SASL for kerberos.

...

This feature requires some co-operation between the socket server and the api layer. The API layer will handle the authenticate request, but the username will be associated with the connection. One approach to implementing this would be to add the concept of a Session object that is maintained with the connection and contains the username. The session would be stored in the context for the socket in socket server and destroyed as part of socket close. The session would be passed down to the API layer with each request and we would have something like session.authenticatedAs() to get the username to use for authorization purposes.

Integrity and Encryption

SASL supports authentication alone, authentication + integrity protection (signing), and authentication + integrity & encryption. I think TLS has a similar set of options.

Integrity protection and encryption require actually translating the bytes being transmitted on a per client basis. These will not work with the sendfile optimization and we would have to disable this optimization for requests with this more stringent security requirement.

Authorization

The plan will be to support unix-like permissions on a per-topic level.

...

Note that the PermissionManager api deals with whatever notion of groups or acls internally. So if via some group mechanism we have assigned the READ permission to an entire group we still do the check at the user level and internally this api needs to resolve the permission at the group level.

Open Questions

• On-the-wire encryption: do we need to do this? If so we will have to disable the sendfile optimization when encryption is used.
• Groups vs ACLs: need to understand pros and cons.
• Can we do everything over a single port?