Status

Current state: ["DISCUSSION"]. Accepted

Discussion thread: here

JIRA: KAFKA-1696 

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

...

getDelegationToken(request: DelegationTokenRequestCreateDelegationTokenRequest):  CreateDelegationTokenResponse

class DelegationTokenRequestCreateDelegationTokenRequest(renewer: Set[KafkaPrincipal] = Set.empty, maxLifeTime: long = -1)

class DelegationTokenResponseCreateDelegationTokenResponse(ownerissueTimeMillis:  KafkaPrincipallong,  expiryTimeMillisexpiryTimeMillis: long, renewer: Set[KafkaPrincipal], maxLifeTime:  maxLifeTime: long, tokenId: String, hmac: byte[])  

renewDelegationToken(request: RenewDelegationTokenRequest):  DelegationTokenResponseRenewDelegationTokenResponse

class RenewDelegationTokenRequest(hmac: byte[], expiryTimeMillis: long) 

expireToken(request: ExpireTokenRequestExpireDelegationTokenRequest): ExpireDelegationTokenResponse

class ExpireTokenRequestExpireDelegationTokenRequest(hmac: byte[], expireAt: long  = Systemtime.currentTimeMillis) 

Protocol changes

DelegationTokenRequest

describeToken(request: DescribeDelegationTokenRequest): DescribeDelegationTokenResponse

class DescribeDelegationTokenRequest(owner: Set[KafkaPrincipal] ) 

Protocol changes

CreateDelegationTokenRequest

CreateDelegationTokenRequest

DelegationTokenRequest => [Renewer] MaxDateMs
  Renewer => string
  MaxDateMs => INT64

Renewer

MaxDateMs

DelegationTokenResponse

CreateDelegationTokenResponse

DelegationTokenResponseCreateDelegationTokenResponse => ErrorCode TokenDetails
  ErrorCode => INT16
  TokenDetails =>  IssueDateMs ExpiryDateMs MaxDateMs TokenId HMAC 
    IssueDateMs  => INT64
    ExpiryDateMs => INT64
    MaxDateMs => INT64	
    TokenId => String 
    HMAC => bytes

IssueDateMs

MaxDateMs

ExpiryDateMs

TokenId

TokenId

HMAC

...

RenewPeriodMs

RenewDelegationTokenResponse

RenewDelegationTokenResponse

RenewDelegationTokenResponse => 

RenewDelegationTokenResponse => ErrorCode 
   ErrorCode => INT32 
   ExpiryDateMs => INT64

...

* TokenNotFoundException 

ExpireTokenRequest

ExpireDelegationTokenRequest

ExpireTokenRequestExpireDelegationTokenRequest => HMAC expiryDateMs
  HMAC => bytes
  ExpiryDateMs => INT64

ExpiryDateMs

...

ExpireDelegationTokenResponse

ExpireDelegationTokenResponse

ExpireTokenResponse => ErrorCode
  ErroCode => INT32
  ExpiryDateMs => INT64

ErrorCode

ExpiryDateMs

Possible Error Codes

* DelegationTokenDisabledException

...

* TokenNotFoundException 

DescribeTokenRequest

DescribeDelegationTokenRequest

DescribeTokenRequestDescribeDelegationTokenRequest => [Owner]
    Owner => String

ErrorCode

Owner

...

DescribeDelegationTokenResponse

DescribeTokenResponseDescribeDelegationTokenResponse => ErrorCode [TokenDetails]
  ErrorCode => INT16
  TokenDetails => Owner IssueDateMs ExpiryDateMs TokenId HMAC [Renewer]
    Owner => String
    IssueDateMs  => INT64
    ExpiryDateMs => INT64
    TokenIdMaxDateMs => String INT64	
    HMACTokenId => bytesString 
    HMAC => bytes
    Renewer => String

Owner

IssueDateMs

MaxDateMs

ExpiryDateMs

TokenId

HMAC

Renewer

...

1. delegation.token.max.lifetime.ms : The token has a maximum lifetime beyond which it cannot be renewed anymore. Default value 7 days.

2. delegation.token.expiry.time.ms : The token validity time in seconds before the token needs to be renewed. Default value 1 day.

3. delegation.token.master.key : masterKey/secret to generate and verify delegation tokens. This masterKey/secret needs to be configured with all the brokers. If  Same secret key must be configured across all the brokers.  If the masterKey/secret is not set or set to empty string, brokers will disable the delegation token support.

...

...

The MasterKey/secret is used to generate and verify delegation tokens. This is supplied using config option. This Same secret needs to key must be configured with across all the brokers. The current proposal does not support rotation of secret. If  If the secret is not set or set to empty string, brokers will disable the delegation token support. The current proposal does not support rotation of secret.

Procedure to manually rotate the secret:

We require a re-deployment when the secret needs to be rotated. During this process, already connected clients will continue to work. But any new connection requests and renew/expire requests with old tokens can fail.

• expire all existing tokens
• rotate the secret by rolling upgrade, and
• generate new tokens

Token acquisition

Following steps describe how tokens can be acquired:

...

Token is stored in Zookeeper as properties in the path /tokenauthdelegation_token/tokens/<tokenUID>. S  During server startup and token creation, scram credentials are generated during token creation. We will use password=HMAC, iterations=4096 for scram credential generation.and stored in memory (TokenCache).  

//Delegation Token Details for tokenID token123: Zookeeper persistence path /tokenauth/tokens/token123
{
   "version":1,
   "owner" : "owner",
   "renewer" : "renewer",
   "issueDateissueTimestamp" : "issueDateissueTimestamp",
   "tokenIDmaxTimestamp" : "sequence-numbermaxTimestamp",
   //Store SCRAM credentials as per KIP-84
 "expiryTimestamp" : "expiryTimestamp",
   "credentialstokenID" : {
          "SCRAM-SHA-512" : "salt=djR5dXdtZGNqamVpeml6NGhiZmMwY3hrbg==,stored_key=sb5jkqStV9RwPVTGxG1ZJHxF89bqjsD1jT4S...==,server_key=...,iterations=4096",
          "SCRAM-SHA-256" : "salt=10ibs0z7xzlu6w5ns0n188sis5,stored_key=+Acl/wi1vLZ95Uqj8rRHVcSp6qrdfQIwZbaZBwM0yvo=,server_key=nN+fZauE6vG0hmFAEj/49+2yk0803y67WSXMYkgh77k=,iterations=4096"
    }
};

SCRAM Extensions

SCRAM messages have an optional extensions field which is a comma-separated list of key=value pairs.
After KIP-84 implementation , an extension will be added to the first client SCRAM message to indicate
that authentication is being requested for a delegation token. This will enable Kafka broker to obtain
credentials and principal using a different code path for delegation tokens.

JAAS configuration

KafkaClient {
	org.apache.kafka.common.security.scram.ScramLoginModule required
	username="test123"
	password="ab24267ac3e827e00e57cdf98465baccecbbeced512e90a719026177e04e547e";
    tokenauth=true
};

DelegationToken Client

We will be providing a DelegationToken Client using which users can generate, renew and expire the tokens. This may part of AdminClient implementation  (KIP-4).

"UUID",
};

SCRAM Extensions

SCRAM messages have an optional extensions field which is a comma-separated list of key=value pairs.
After KIP-84 implementation , an extension will be added to the first client SCRAM message to indicate
that authentication is being requested for a delegation token. This will enable Kafka broker to obtain
credentials and principal using a different code path for delegation tokens.

JAAS configuration

KafkaClient {
	org.apache.kafka.common.security.scram.ScramLoginModule required
	username="test123"
	password="ab24267ac3e827e00e57cdf98465baccecbbeced512e90a719026177e04e547e";
    tokenauth=true
};

DelegationToken Client

We will be providing a DelegationToken Client using which users can generate, renew and expire the tokens. This may part of AdminClient implementation  (KIP-4).

public class DelegationTokenClient  {

    public TokenDetails generateToken(List<String> renewers, long maxLifeTime);

    public boolean renewToken(bytes[] hmac,  long renewPeriod);

    public boolean expireToken(bytes[] hmac, long expireTimeStamp);

public class DelegationTokenClient  {

    public TokenDetailsboolean generateToken(List<String> renewers, long maxLifeTime);

    public boolean renewToken(bytesinvalidateToken(bytes[] hmac,  long renewPeriod);
 
    public booleanList<TokenDetails> expireToken(bytes[] hmac, long expireTimeStampdescribeTokens();

    public booleanvoid invalidateTokenclose(bytes[] hmac);
 
    public List<TokenDetails> describeTokens();

    public void close();

}

Command line tool

 We will provide a CLI to acquire delegation tokens, renew tokens and to invalidate/expire tokens. 

Changes to Java Clients (producer/consumer)

KIP-85 allows dynamic JAAS configuration for Kafka clients. After this we can easily configure the
delegation token for SCRAM-SHA-256 authentication.

Below diagram shows the steps required to use the delegation tokens.

...

}

Command line tool

We will provide a commandline script to acquire delegation tokens, renew tokens, invalidate/expire and to describe tokens. 

bin/kafka-delegation-token.sh --bootstrap-server broker1:9092 --create --renewer renewer1,renewer2 --max-life-time 1486750745585
bin/kafka-delegation-token.sh --bootstrap-server broker1:9092 --renew --hmac hmacString --renew-time-period 50745585
bin/kafka-delegation-token.sh --bootstrap-server broker1:9092 --expire --hmac hmacString --expiry-time-period 50745585
bin/kafka-delegation-token.sh --bootstrap-server broker1:9092 --describe --owner owner1,owner2

Changes to Java Clients (producer/consumer)

KIP-85 allows dynamic JAAS configuration for Kafka clients. After this we can easily configure the
delegation token for SCRAM-SHA-256 authentication.

Below diagram shows the steps required to use the delegation tokens.

Image Added

ACL

Currently, we only allow a user to create delegation token for that user only. Renew and expire requests should come from owner or renewers of the token. So we don't need ACLs for create/renew/expire requests.  

Owners and the renewers can always describe their own tokens. To describe others tokens, we can add DESCRIBE operation on Token Resource.  In future, when we extend the support to allow a user to acquire delegation tokens for other users, then we can enable CREATE/DELETE operations.

DESCRIBE

CREATE

DELETE

Q/A

Q1. Is there any dependency on Hadoop APIs/Libraries?

...