THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
Problem
We want to secure BookKeeper clusters, so at least we have to cover this main issues:
- SSL/TLS support for encrypting comunications and/or to implement client authentication
- Use ACLs while writing to ZooKeeper
- Provide an extensible Authentication framework (in BookKeeper 4.4.0 there is already partial support for this)
- Provide a out-of-the box plugin which implements standard SASL authentication, with at least support for GSSAPI/Kerberos and DIGEST-MD5 mechs
- Provide support for Authorization
Proposal
Major concers are about protocol compatibility, configuration on Bookie side and client-side and rolling upgrades.
SSL Support
TBD
Secure ZooKeeper communications
In order to implement secure ZooKeeper comunications at least we have to set ACLs which prevent unknown users to modify and/or access ledgers metadata and bookies metadata
The idea is to have a global "secure zookeeper comunications", to be set both on bookies and on clients (as bookkeeper clients do the most of the work actually).
Initial proposal is to set the default ZooKeeper ACL ZooDefs.Ids.CREATOR_ALL_ACL to every Z-Node created from BookKeeper clients and Bookies.
This will work if every client and bookie performs authentication to ZooKeeper with the same user/principal. This is very common in the Hadoop ecosystem (see Kafka, HBase...) and with Kerberos bases setups.
Usually is such setups each machine logs in using a principal in the form zkclient/HOSTNAME@REALM and the HOSTNAME and REALM part are stripped out, this way each client gets authenticated as simply "zkclient" user.
Rolling upgrade to switch from authentication types:
The proposal is to support the switch from "no-auth" to any plugin is the following:
1) when no auth is configured on bookie-side the default "AuthDisabledPlugin" plugin comes to play
2) when no auth is configured on bookies the bookie goes to a permissive mode in which any client is allowed to connect and use bookie functions, any 'auth' message will be answered with an OK status code from the AuthDisabledPlugin
3) when no auth is configured on bookies each attempt of authentication by the client will receive the OK status code from the AuthDisabledPlugin, and so the client will understand that the bookie is not performing authentication, and goes to the authenticated state
A rolling upgrade from AuthDisabledPlugin to any other plugin can be implemented by rebooting clients with the new plugin: this way bookies will authenticate clients with AuthDisabledPlugin and clients will go to authenticated state anyway.
Then you will restart bookies with the new plugin and clients will perform real authentication.
Bookie-to-bookie comunications will follow the same scheme during the rolling upgrade, because each bookie can be a client for the other bookies and will authenticate thru the AuthDisabledPlugin flow or to the requested plugin flow.
Attaching the authenticated user to the connection:
Another issue will be to attach to the server-side view of the connection the user-id (principal) of the client which performed authentication, this will be the base for ACLs.
A change to the AuthProvider interface will be needed.
ISSUE: Do we have to add this feature on 4.5.0 as we are breaking compatibility with 4.4 on protocol ?
SASL/Kerberos support:
The initial proposal is to support SASL authentiaction as ZooKeeper does, this is turn is implemented using standard Java support for JAAS and SASL
The admin need to simply write a JAAS configuration file like this for Kerberos and setup keytabs for each machine.
Bookie {
com.sun.security.auth.module.Krb5LoginModule required debug=true
useKeyTab=true
keyTab=/path/to/server.keytab
storeKey=true
useTicketCache=false
principal=bookkeeper/HOSTNAME@REALM
};
BookKeeper {
com.sun.security.auth.module.Krb5LoginModule required debug=true
useKeyTab=true
keyTab=/path/to/client.keytab
storeKey=true
useTicketCache=false
principal=username/HOSTNAME@REALM
};
For simple DIGEST-MD5 login we can use the same LoginModule class as ZooKeeper and implement a similar handler
Bookie {
org.apache.zookeeper.server.auth.DigestLoginModule required
user_user1="testpwd";
user_user2="testpwd";
};
BookKeeper {
org.apache.zookeeper.server.auth.DigestLoginModule required
username="user1"
password="testpwd";
};
Following the conventions the "Bookie" principal will be bookeeper/HOSTNAME@REALM (for instance in zookeeper it has to be zookeeper/HOSTNAME@REALM and for kafka it is kafka/HOSTNAME@REALM).
For simple authentication we do not care about dealing with principal manipulations, like stripping out HOSTNAME and REALM, as it it possible in ZooKeeper (see kerberos.removeHostFromPrincipal, kerberos.removeRealmFromPrincipal properties).
In BookKeeper bookie-to-bookie comunication is simply a client-to-bookie comunication and so on Bookie the admin will need to configure both the server-side configuration entry (Bookie) and the client-side configuration entry (BookKeeper).
ISSUE: On ZookKeeper the SASL mech is decided upon the type of JAAS Subject, this is very simply from admin to be configured. Should we make the configuration more explicit and make the admin configure that the system will run using a specific mech and fail if this mech is not configured at JAAS level ? Something like a configuration property sasl.mech=GSSAPI
Action