...

Current state: ["DISCUSSION"]. 

Discussion thread: here

JIRA: KAFKA-1696 

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

...

class DelegationTokenRequest(renewer: option(Set[KafkaPrincipal) ] = NoneSet.empty, maxLifeTime: long = -1)

class DelegationTokenResponse(owner:  KafkaPrincipal,  expiryTimeMillis: long, renewer: Set([KafkaPrincipal) = None], maxLifeTime: long = -1, tokenId: String, hmac: byte[])  

...

• A client connects with one of the kafka broker. Client must be authenticated using any of the available secure channels so it must have a way to authenticate, i.e. Kerberos keytab or TGT.

• Once a client is authenticated, it will make a broker side call to issue a delegation token.  The request for delegation token will have to contain an optional renewer identity and max lifetime for token. The renewer is the user that is allowed to renew this token before the max lifetime expires. Renewer will default to the owner if not provided and Max life time will default to a server side config value (default  days) Brokers will allow a token to be renewed for ever if no value is provided until maxLifeTime but a token will still expire if not renewed by the expiry time. The expiry time will be a broker side configuration and will default to min (24 hours, max lifetimemaxlifeTime) . A Delegation Token request can be represented as class DelegationTokenRequest(renewer: option(KafkaPrincipal) = NoneSet[KafkaPrincipal], maxLifeTime: long = -1). The owner is implicit in the request connection as the user who requested the delegation token.

• The broker generates a shared secret based on HMAC-SHA256SASM(a Password/Secret shared between all brokers, randomly generated tokenId). We can represent a token as scala case class DelegationToken(owner: KafkaPrincipal, renewer: Set[KafkaPrincipal], maxLifeTime: long, id: String, hmac: String, expirationTime: long)

• Broker stores this token in its in memory cache. Broker also stores the DelegationToken without the hmac in the zookeeper. As all brokers share the Password/Secret to generate the HMAC-SHA256SASM, they can read the request info from zookeeper , generate the hmac and store the delegation token in local cache.

• All brokers will have a cache backed by zookeeper so they will all get notified whenever a new token is generated and they will update their local cache whenever token state changes.

• Broker returns the token to Client. Client is expected to only make delegation token request over an encrypted channel so the token in encrypted over the wire.

• Client is free to distribute this token to other clients. It is the client’s responsibility to distribute the token securely.

...

 We will reuse the current SASL channel but for authentication using delegation we will use DIGEST-MD5delegation token based authentication.

• Client will already have the delegation token which it will present during the authentication phase.

• Server will look up the token from its token cache, if it finds a match and token is not expired it will authenticate the client and the identity will be established as the owner of the delegation token.

• If the token is not matched or token is expired, broker throws appropriate exception back and does not allow the client to continue.

...

• The client authenticates using Kerberos or any other available authentication scheme. ( Can this authentication be done using delegation token? if it is allowed then we probably do not want to default renewer as owner as anyone with delegation token can actually renew their own token forever. Instead if no renewer is provided we should mark those tokens as non renewable or we should make renewer a mandatory request field during token acquisition and ensure renewer can not be set to owner)A token can not be renewed if the initial authentication is done through delegation token, client must use a different auth scheme. 

• Client sends a request to renew a token with an optional renew life time which must be < max life time of token.

• Broker looks up the token, if token is expired or if the renewer’s identity does not match with the token’s renewerrenewers, or if token renewal is beyond the Max life time of token,  broker disallows the operation by throwing an AuthorizationException with appropriate messageexception.

• If none of the above conditions are matched, broker updates token’s expiry. Note that the HMAC-SHA256 SASM is unchanged so the token on client side is unchanged. Broker updates the expiration in its local cache and on zookeeper so other brokers also get notified and their cache statuses are updated as well.

...

 We will provide a CLI to acquire delegation tokens, renew tokens and to invalidate/expire tokens. 

Alternatives

Originally we considered to not have any shared Secret at config level. This required us to chose one of the 2 3 options:

• Let each broker generate a Random secret on each acquisition request and use this secret to generate the hmac. Broker will store the hmac and secret in zookeeper. However as zkClient does not support SSL the hmac will be on wire unencrypted which is not safe.
• Use controller instead of zookeeper as the central location where tokens are generated,renewed and distributed from. You can review the discussion and pro/con here.
• Use controller to generate and rotate secret and distribute it to all brokers. Brokers will generate hmac based on *current* secret. The advantage is secret rotation can be more frequent and automated. The disadvantage is added complexity as new controller level APIs need to be created to push/pull tokens from controller to brokers and brokers needs to keep a list of valid secrets till max( max life time of all tokens).

...