Table of Contents |
---|
Status
Current state: ["DISCUSSION"]. Accepted
Discussion thread: here
JIRA: KAFKA-1696
...
getDelegationToken(request: DelegationTokenRequestCreateDelegationTokenRequest): CreateDelegationTokenResponse
class DelegationTokenRequestCreateDelegationTokenRequest(renewer: Set[KafkaPrincipal] = Set.empty, maxLifeTime: long = -1)
class DelegationTokenResponseCreateDelegationTokenResponse(expiryTimeMillisissueTimeMillis: long, renewer: Set[KafkaPrincipal], maxLifeTimeexpiryTimeMillis: long, maxLifeTime: long, tokenId: String, hmac: byte[])
renewDelegationToken(request: RenewDelegationTokenRequest): RenewRenewDelegationTokenResponse
class RenewDelegationTokenRequest(hmac: byte[], expiryTimeMillis: long)
expireToken(request: ExpireTokenRequestExpireDelegationTokenRequest): ExpireTokenResponseExpireDelegationTokenResponse
class ExpireTokenRequestExpireDelegationTokenRequest(hmac: byte[], expireAt: long = Systemtime.currentTimeMillis)
describeToken(request: DescribeTokenRequestDescribeDelegationTokenRequest): DescribeTokenRespopnseDescribeDelegationTokenResponse
class DescribeDescribeDelegationTokenRequest(owner: Set[KafkaPrincipal] )
Protocol changes
DelegationTokenRequest
CreateDelegationTokenRequest
Code Block |
---|
DelegationTokenRequestCreateDelegationTokenRequest => [Renewer] MaxDateMs Renewer => string MaxDateMs => INT64 |
Field | Description |
---|---|
Renewer | Renewer is an Kafka PrincipalType+name string, who is allowed to renew this token before the max lifetime expires. If Renewer list is empty, then Renewer will default to the owner (Principal which requested this token). |
MaxDateMs | Max lifetime for the token in milliseconds. If the value is -1, then MaxLifeTime will default to a server side config value (delegation.token.max.lifetime.ms). |
DelegationTokenResponse
CreateDelegationTokenResponse
Code Block |
---|
DelegationTokenResponseCreateDelegationTokenResponse => ErrorCode TokenDetails ErrorCode => INT16 TokenDetails => IssueDateMs ExpiryDateMs MaxDateMs TokenId HMAC IssueDateMs => INT64 ExpiryDateMs => INT64 TokenIdMaxDateMs => String INT64 HMACTokenId => bytesString HMAC => bytes |
Field | Description |
---|---|
IssueDateMs | timestamp (in msec) when this token was generated. Unit is milliseconds since the beginning of the epoch (midnight Jan 1, 1970 (UTC)). ExpiryDateMs |
MaxDateMs | timestamp (in msec) at which this token expires. Unit is milliseconds since the beginning of the epoch (midnight Jan 1, 1970 (UTC)). |
ExpiryDateMs | max life timestamp (in msec) of this token. Unit is milliseconds since the beginning of the epoch (midnight Jan 1, 1970 (UTC)). |
TokenId | Sequence number to ensure uniqueness |
HMAC | Keyed-hash message authentication code |
...
* TokenNotFoundException
ExpireTokenRequest
ExpireDelegationTokenRequest
Code Block |
---|
ExpireTokenRequestExpireDelegationTokenRequest => HMAC expiryDateMs HMAC => bytes ExpiryDateMs => INT64 |
Field | Description |
---|---|
HMAC | HMAC of the delegation token to be renewed |
ExpiryDateMs | Expiry time period in milliseconds. If the value is -1, then the token will get invalidated immediately. |
...
ExpireDelegationTokenResponse
Code Block |
---|
ExpireTokenResponseExpireDelegationTokenResponse => ErrorCode ErroCode => INT32 ExpiryDateMs => INT64 |
...
* TokenNotFoundException
DescribeTokenRequest
DescribeDelegationTokenRequest
Code Block |
---|
DescribeTokenRequestDescribeDelegationTokenRequest => [Owner] Owner => String |
Field | Description |
---|---|
ErrorCode | |
Owner | Kakfa Principal which requested the delegation token. If the Owner list is null (i.e., length is -1), the response contains all the allowed tokens from all owners |
...
. If Owner list is empty, the response is empty list. |
DescribeDelegationTokenResponse
Code Block |
---|
DescribeTokenResponseDescribeDelegationTokenResponse => ErrorCode [TokenDetails] ErrorCode => INT16 TokenDetails => Owner IssueDateMs ExpiryDateMs TokenId HMAC [Renewer] Owner => String IssueDateMs => INT64 ExpiryDateMs => INT64 MaxDateMs => INT64 TokenId => String HMAC => bytes Renewer => String |
Field | Description | ||
---|---|---|---|
Owner | Kakfa Principal which requested the delegation token | ||
IssueDateMs | timestamp (in msec) when this token was generated. Unit is milliseconds since the beginning of the epoch (midnight Jan 1, 1970 (UTC)). ExpiryDateMs | ||
MaxDateMs | max life timestamp (in msec) | at whichof this token | expires. Unit is milliseconds since the beginning of the epoch (midnight Jan 1, 1970 (UTC)). |
TokenId | Sequence number to ensure uniqueness | ||
ExpiryDateMs | timestamp (in msec) at which this token expires. Unit is milliseconds since the beginning of the epoch (midnight Jan 1, 1970 (UTC)). | ||
TokenId | Sequence number to ensure uniqueness | ||
HMAC | Keyed-hash | HMAC | Keyed-hash message authentication code |
Renewer | Renewers list |
...
TokenDetails:
- Owner ID -- Username that this token will authenticate as
- Renewers ID -- designated renewers list
- Issue date -- timestamp (in msec) when this token was generated
- Expiry date -- timestamp (in msec) at which this token expires
- Max Date - max life timestamp (in msec) of this token.
- TokenID – UUID to TokenID -- Sequence number to ensure uniqueness
TokenAuthenticator(HMAC) := HMAC_SHA1(master key, TokenID)
Authentication Token := (TokenDetails, TokenAuthenticator(HMAC))
...
The MasterKey/secret is used to generate and verify delegation tokens. This is supplied using config option. Same secret key must be configured across all the brokers. The current proposal does not support rotation of secret. If the secret If the secret is not set or set to empty string, brokers will disable the delegation token support. The current proposal does not support rotation of secret.
Procedure to manually rotate the secret:
We require a re-deployment when the secret needs to be rotated. During this process, already connected clients will continue to work.
Token acquisition
But any new connection requests and renew/expire requests with old tokens can fail.
- expire all existing tokens
- rotate the secret by rolling upgrade, and
- generate new tokens
Token acquisition
Following steps describe how tokens can be acquired:
...
Token is stored in Zookeeper as properties in the path /tokenauthdelegation_token/tokens/<tokenUID>. S During server startup and token creation, scram credentials are generated during token creation. We will use password=HMAC, iterations=4096 for scram credential generation.and stored in memory (TokenCache).
Code Block | ||||
---|---|---|---|---|
| ||||
//Delegation Token Details for tokenID token123: Zookeeper persistence path /tokenauth/tokens/token123 { "version":1, "owner" : "owner", "renewer" : "renewer", "issueDateissueTimestamp" : "issueDateissueTimestamp", "tokenIDmaxTimestamp" : "sequence-numbermaxTimestamp", //Store SCRAM credentials as per KIP-84 "credentials": { "expiryTimestamp" : "expiryTimestamp", "SCRAM-SHA-512tokenID" : "salt=djR5dXdtZGNqamVpeml6NGhiZmMwY3hrbg==,stored_key=sb5jkqStV9RwPVTGxG1ZJHxF89bqjsD1jT4S...==,server_key=...,iterations=4096", "SCRAM-SHA-256" : "salt=10ibs0z7xzlu6w5ns0n188sis5,stored_key=+Acl/wi1vLZ95Uqj8rRHVcSp6qrdfQIwZbaZBwM0yvo=,server_key=nN+fZauE6vG0hmFAEj/49+2yk0803y67WSXMYkgh77k=,iterations=4096" } }; |
SCRAM Extensions
UUID",
}; |
SCRAM Extensions
SCRAM messages have an optional extensions field which is a comma-separated list of key=value pairs.
After KIP-84 implementation , an extension will be added to the first client SCRAM message to SCRAM messages have an optional extensions field which is a comma-separated list of key=value pairs.
After KIP-84 implementation , an extension will be added to the first client SCRAM message to indicate
that authentication is being requested for a delegation token. This will enable Kafka broker to obtain
credentials and principal using a different code path for delegation tokens.
...
Code Block | ||||
---|---|---|---|---|
| ||||
public class DelegationTokenClient {
public TokenDetails generateToken(List<String> renewers, long maxLifeTime);
public boolean renewToken(bytes[] hmac, long renewPeriod);
public boolean expireToken(bytes[] hmac, long expireTimeStamp);
public boolean invalidateToken(bytes[] hmac);
public List<TokenDetails> describeTokens();
public void close();
} |
Command line tool
;
} |
Command line tool
We will provide a commandline script to acquire delegation tokens, renew tokens, invalidate/expire and to describe tokens.
bin/kafka-delegation-token.sh --bootstrap-server broker1:9092 --create --renewer renewer1,renewer2 --max-life-time 1486750745585
bin/kafka-delegation-token.sh --bootstrap-server broker1:9092 --renew --hmac hmacString --renew-time-period 50745585We will provide a commandline script to acquire delegation tokens, renew tokens, invalidate/expire and to describe tokens.
bin/kafka-delegation-token.sh --bootstrap-server broker1:9092 --create expire --renewer renewer1,renewer2 hmac hmacString --maxexpiry-lifetime-time 1486750745585period 50745585
bin/kafka-delegation-token.sh --bootstrap-server broker1:9092 --renew describe --hmac hmacString --renew-time-period 50745585
bin/kafka-delegation-token.sh --bootstrap-server broker1:9092 --expire --hmac hmacString --expiry-time-period 50745585
bin/kafka-delegation-token.sh --bootstrap-server broker1:9092 --describe --owner owner1,owner2
Changes to Java Clients (producer/consumer)
KIP-85 allows dynamic JAAS configuration for Kafka clients. After this we can easily configure the
delegation token for SCRAM-SHA-256 authentication.
Below diagram shows the steps required to use the delegation tokens.
...
owner owner1,owner2
Changes to Java Clients (producer/consumer)
KIP-85 allows dynamic JAAS configuration for Kafka clients. After this we can easily configure the
delegation token for SCRAM-SHA-256 authentication.
Below diagram shows the steps required to use the delegation tokens.
ACL
Currently, we only allow a user to create delegation token for that user only. Renew and expire requests should come from owner or renewers of the token. So we don't need ACLs for create/renew/expire requests.
Owners and the renewers can always describe their own tokens. To describe others tokens, we can add DESCRIBE operation on Token Resource. In future, when we extend the support to allow a user to acquire delegation tokens for other users, then we can enable CREATE/DELETE operations.
Operation | Resource | API |
---|---|---|
DESCRIBE | Token | describeTokens |
CREATE | Token | createToken (Will be introduced in a future release) |
DELETE | Token | deleteToken (Will be introduced in a future release) |
Q/A
Q1. Is there any dependency on Hadoop APIs/Libraries?
...