...

Major concers are about protocol compatibility, configuration on Bookie side and client-side and rolling upgrades.

SSL Support

TBD

Secure ZooKeeper

...

data structures

In order to implement secure a secured ZooKeeper comunications layout at least we have to set ACLs which prevent unknown users to modify and/or access ledgers metadata and bookies metadata.

The idea is to have a global "secure zookeeper comunicationsnodes" (zkEnableSecurity), to be set both on bookies and on clients (, as bookkeeper clients do the most of the work actually).

Initial The initial proposal is to set the default ZooKeeper ACL ZooDefs.Ids.CREATOR_ALL_ACL to every Z-Node created from BookKeeper clients and Bookies.

This will work fine only if every client and bookie performs authentication to ZooKeeper with the same "user/principal".

This is a very common practice in the Hadoop ecosystem (see Kafka, HBase...) and with Kerberos bases based setups.

Usually is such setups in a Kerberized ZooKeeper setup each machine logs in using a principal in the form zkclient/HOSTNAME@REALM and ZooKeeper strips out the HOSTNAME and REALM part are stripped out, : this way each client gets authenticated as simply "zkclient" user.

Changes on protocol for AuthMessage message:

On 4.4.0 there is a issue on the BookieAuthProvider and ClientAuthProvider classes, these classes depend directly on the AuthMessage class which in turn depends on protobuf, which is a shaded dependency.

The idea is to remove the usage of that class on the public API and make the implementor use a standard AuthToken class which is a simple wrapper for an array of bytes.

Common Authentication frameworks, like SASL, need only to exchange a sequence of arrays of bytes, and so this new API would be suitable for most of the usecases.

On 4.4.0 the AuthMessage message is defined to be extensible, such a feature will not be needed any more.

ISSUE: can we drop the 'extensions' line on the definition of AuthMessage ?

Rolling upgrade to switch from authentication types:

...

Then you will restart bookies with the new plugin and clients will perform real authentication.

Bookie-to-bookie Bookie comunications will follow the same scheme flow during the rolling upgrade, because each bookie can be a client for the other bookies and will authenticate thru the AuthDisabledPlugin flow or to the requested plugin flow.

Attaching the authenticated user to the

...

server-side peer:

Another issue will be is to attach to the server-side view of the connection (there is no specific class, it is the BookiePipeline instance created for each client)  the user-id (principal) of the client which performed authentication, this will be the base for ACLs, quotas, auditing and further security-related features.

A change to the AuthProvider interface will be needed, at least the AuthHandshakeCompleteCallback needs to receive the user-id upon successful authentication.

from:

class AuthHandshakeCompleteCallback implements GenericCallback<Void> {
  @Override
  public void operationComplete(int rc, Void v) {
    if (rc == BKException.Code.OK) {
       authenticated = true;
    } else {
        LOG.debug("Authentication failed on server side");
    }
    }
}

to something like:

class AuthHandshakeCompleteCallback implements GenericCallback<BookKeeperPrincipal> {
  @Override
  public void operationComplete(int rc, BookKeeperPrincipal userId) {
    if (rc == BKException.Code.OK) {
       authenticated = true;
       authorizedId = userId;
    } else {
        LOG.debug("Authentication failed on server side");
    }
    }
}

The "authorizedId" field of AuthHandler.ServerSideHandler need to be pushed back to the BookiePipeline.

Thinking about SSL mutual autehntication it wold be useful to change the BookieAuthProvider.Factory#newProvider method.

In 4.4.0 this method receives only the InetSocketAddress of the client connection, it would be better to pass a structure contains:

• the InetSocketAddress of the client (maybe it would be a SocketAddress, in order to deal better with "Local" connections)
• Certificates and other kind of Principals attached to the connection

It is better not to hard-code the dependency on Netty on such an interface, it would be enough to create a simple bean for 4.5.0 like:

class PeerInfo {
     SocketAddress address;
     List<Object> principals;
}

It would be better ISSUE: Do we have to add this feature on 4.5.0  as we are breaking compatibility with 4.4 on protocol ?

Authentication framework

The final signature would be:

 BookieAuthProvider newProvider(PeerInfo peerInfo,
                                       GenericCallback<BookKeeperPrincipal> completeCb);

ISSUE: do we need a  BookKeeperPrincipal class or it is enough to use a simple java.lang.String ?

Some projects, like Kafka use a common class (KakfaPrincipal, see KIP-11 - Authorization Interface)  in order to provide better support for the other features

SASL/Kerberos support:

The initial proposal is to support SASL authentiaction as ZooKeeper does, this is turn is implemented using standard Java support for JAAS and SASL

We can implement an AuthPlugin which uses SASL (see https://github.com/eolivelli/bookkeeper/tree/BOOKKEEPER-391/bookkeeper-server/src/main/java/org/apache/bookkeeper/sasl as a proof-of-concept implementation) and bundle it on standard BookKeeper distribution.

The admin need needs to simply write  a JAAS configuration file like this for Kerberos and setup keytabs for each machine.

Bookie {
       com.sun.security.auth.module.Krb5LoginModule required debug=true
       useKeyTab=true
       keyTab=/path/to/server.keytab
       storeKey=true
       useTicketCache=false
       principal=bookkeeper/HOSTNAME@REALM
};

BookKeeper {
       com.sun.security.auth.module.Krb5LoginModule required debug=true
       useKeyTab=true
       keyTab=/path/to/client.keytab
       storeKey=true
       useTicketCache=false
       principal=username/HOSTNAME@REALM
};

Auditor {
       com.sun.security.auth.module.Krb5LoginModule required debug=true
       useKeyTab=true
       keyTab=/path/to/server.keytab
       storeKey=true
       useTicketCache=false
       principal=bookkeeper/HOSTNAME@REALM
};

ZookKeeper enable the usage of JAAS to configure SASL based DIGEST-MD5 authentication, this is very useful for little deployments and for testing purposes. See ZooKeeper documentation for better explanations. Kakfa supports a similar configuration for simple JAAS/SASL setups.

For simple DIGEST-MD5 login we can use  the same LoginModule class as ZooKeeper and implement a similar handlerZooKeeper uses JAAS files like these:

Bookie {       
       org.apache.zookeeper.server.auth.DigestLoginModule required
       user_user1="testpwd";
       user_user2="testpwd";
       user_auditor="testpwd";
 };
 
BookKeeper {
       org.apache.zookeeper.server.auth.DigestLoginModule required
       username="user1"
       password="testpwd";
};

Auditor {
       org.apache.zookeeper.server.auth.DigestLoginModule required
       username="auditor"
       password="testpwd";
};

Following the conventions the "Bookie" principal will be bookeeper/HOSTNAME@REALM (for instance in zookeeper it has to be zookeeper/HOSTNAME@REALM and for kafka it is kafka/HOSTNAME@REALM).

For simple authentication we do not care about dealing with principal manipulations, like stripping out HOSTNAME and REALM, as it it possible in ZooKeeper The authorized id (user) will be the Principal attached to the JAAS subject, so bookeeper/HOSTNAME@REALM. In ZooKeeper there are options to strip out the HOSTNAME part and the REALM part (see kerberos.removeHostFromPrincipal, kerberos.removeRealmFromPrincipal properties).

ISSUE: In BookKeeper bookie-to-bookie comunication is simply a clientorder to support a distinct JAAS configuration entry for the Auditor (Bookie-to-bookie comunication and so on Bookie the admin will need to configure both the server-side configuration entry (Bookie) and the client-side configuration entry (BookKeeper)Bookie comunications) the special BookKeeper instance used by the Auditor must be configured in order not to use the standard "BookieClient" entry. Maybe we can use an internal configuration option handled by BookKeeper client like 'auditorMode'. In this mode the entry will be choosen following the Auditor conventions.

ISSUE:  On ZookKeeper the SASL mech mechanism is decided upon the type of JAAS Subject, this is very simply from admin to be configured. Should we We should make the configuration more explicit and make the admin configure that the system will run using a specific mech and fail if this mech is not configured at JAAS level ? Something , something like a configuration property sasl.mech=GSSAPI|DIGEST-M5.....

Action

...