Table of Contents |
---|
Status
Current state: ["DISCUSSION"]. Accepted
Discussion thread: here
JIRA: KAFKA-1696
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
...
getDelegationToken(request: DelegationTokenRequestCreateDelegationTokenRequest): CreateDelegationTokenResponse
class DelegationTokenRequestCreateDelegationTokenRequest(renewer: Set[KafkaPrincipal] = Set.empty, maxLifeTime: long = -1)
class DelegationTokenResponseCreateDelegationTokenResponse(ownerissueTimeMillis: KafkaPrincipallong, expiryTimeMillisexpiryTimeMillis: long, renewer: Set[KafkaPrincipal], maxLifeTime: maxLifeTime: long, tokenId: String, hmac: byte[])
renewDelegationToken(request: RenewDelegationTokenRequest): DelegationTokenResponseRenewDelegationTokenResponse
class RenewDelegationTokenRequest(hmac: byte[], expiryTimeMillis: long)
expireToken(request: ExpireTokenRequestExpireDelegationTokenRequest): ExpireDelegationTokenResponse
class ExpireTokenRequestExpireDelegationTokenRequest(hmac: byte[], expireAt: long = Systemtime.currentTimeMillis)
Protocol changes
DelegationTokenRequest
describeToken(request: DescribeDelegationTokenRequest): DescribeDelegationTokenResponse
class DescribeDelegationTokenRequest(owner: Set[KafkaPrincipal] )
Protocol changes
CreateDelegationTokenRequest
Code Block |
---|
CreateDelegationTokenRequest |
Code Block |
DelegationTokenRequest => [Renewer] MaxDateMs
Renewer => string
MaxDateMs => INT64 |
Field | Description |
---|---|
Renewer | Renewer is an Kafka PrincipalType+name string, who is allowed to renew this token before the max lifetime expires. If Renewer list is empty, then Renewer will default to the owner (Principal which requested this token). |
MaxDateMs | Max lifetime for the token in milliseconds. if If the value is -1, then MaxLifeTime will default to a server side config value (delegation.token. |
DelegationTokenResponse
max.lifetime.ms). |
CreateDelegationTokenResponse
Code Block |
---|
DelegationTokenResponseCreateDelegationTokenResponse => ErrorCode TokenDetails ErrorCode => INT16 TokenDetails => IssueDateMs ExpiryDateMs MaxDateMs TokenId HMAC IssueDateMs => INT64 ExpiryDateMs => INT64 MaxDateMs => INT64 TokenId => String HMAC => bytes |
Field | Description | ||
---|---|---|---|
IssueDateMs | timestamp (in msec) when this token was generated. Unit is milliseconds since the beginning of the epoch (midnight Jan 1, 1970 (UTC)). ExpiryDateMs | ||
MaxDateMs | timestamp (in msec) at which this token expires. Unit is milliseconds since the beginning of the epoch (midnight Jan 1, 1970 (UTC)). | ||
ExpiryDateMs | max life timestamp (in msec) of this token. Unit is milliseconds since the beginning of the epoch (midnight Jan 1, 1970 (UTC)). | ||
TokenId | Sequence number to ensure | TokenId | Sequence number to ensure uniqueness |
HMAC | Keyed-hash message authentication code |
...
Field | Description |
---|---|
HMAC | HMAC of the delegation token to be renewed |
RenewPeriodMs | Renew Time period in milliseconds |
RenewDelegationTokenResponse
. If the value is -1, then Renew Time period will default to a server side config value (delegation.token.expiry.time.ms). |
RenewDelegationTokenResponse
Code Block |
---|
RenewDelegationTokenResponse => ErrorCode |
Code Block |
RenewDelegationTokenResponse => ErrorCode
ErrorCode => INT32
ExpiryDateMs => INT64 |
...
* TokenNotFoundException
ExpireTokenRequest
ExpireDelegationTokenRequest
Code Block |
---|
ExpireTokenRequestExpireDelegationTokenRequest => HMAC expiryDateMs HMAC => bytes ExpiryDateMs => INT64 |
Field | Description |
---|---|
HMAC | HMAC of the delegation token to be renewed |
ExpiryDateMs | Expiry Time time period in milliseconds |
...
. If the value is -1, then the token will get invalidated immediately. |
ExpireDelegationTokenResponse
Code Block |
---|
ExpireTokenResponseExpireDelegationTokenResponse => ErrorCode ErroCode => INT32 ExpiryDateMs => INT64 |
Field | Description |
---|---|
ErrorCode | |
ExpiryDateMs | timestamp (in msec) at which this token expires. Unit is milliseconds since the beginning of the epoch (midnight Jan 1, 1970 (UTC)). -1 value will invalidate the token immediately |
...
* TokenNotFoundException
DescribeTokenRequest
DescribeDelegationTokenRequest
Code Block |
---|
DescribeTokenRequestDescribeDelegationTokenRequest => [Owner] Owner => String |
Field | Description |
---|---|
ErrorCode | |
Owner | Kakfa Principal which requested the delegation token. If the Owner list is null (i.e., length is -1), the response contains all the allowed tokens from all owners |
...
. If Owner list is empty, the response is empty list. |
DescribeDelegationTokenResponse
Code Block |
---|
DescribeTokenResponseDescribeDelegationTokenResponse => ErrorCode [TokenDetails] ErrorCode => INT16 TokenDetails => Owner IssueDateMs ExpiryDateMs TokenId HMAC [Renewer] Owner => String IssueDateMs => INT64 ExpiryDateMs => INT64 MaxDateMs => INT64 TokenId => String HMAC => bytes Renewer => String |
Field | Description | ||
---|---|---|---|
Owner | Kakfa Principal which requested the delegation token | ||
IssueDateMs | timestamp (in msec) when this token was generated. Unit is milliseconds since the beginning of the epoch (midnight Jan 1, 1970 (UTC)). ExpiryDateMs | ||
MaxDateMs | max life timestamp (in msec) | at whichof this token | expires. Unit is milliseconds since the beginning of the epoch (midnight Jan 1, 1970 (UTC)). |
TokenId | Sequence number to ensure uniqueness | ||
ExpiryDateMs | timestamp (in msec) at which this token expires. Unit is milliseconds since the beginning of the epoch (midnight Jan 1, 1970 (UTC)). | ||
TokenId | Sequence number to ensure uniqueness | ||
HMAC | Keyed | HMAC | Keyed-hash message authentication code |
Renewer | Renewers list |
...
TokenDetails:
- Owner ID -- Username that this token will authenticate as
- Renewers ID -- designated renewers list
- Issue date -- timestamp (in msec) when this token was generated
- Expiry date -- timestamp (in msec) at which this token expires
- Max Date - max life timestamp (in msec) of this token.
- TokenID – UUID to TokenID -- Sequence number to ensure uniqueness
TokenAuthenticator(HMAC) := HMAC_SHA1(master key, TokenID)
Authentication Token := (TokenDetails, TokenAuthenticator(HMAC))
...
The MasterKey/secret is used to generate and verify delegation tokens. This is supplied using config option. Same secret key must be configured across all the brokers. The current proposal does not support rotation of secret. If the If the secret is not set or set to empty string, brokers will disable the delegation token support. The current proposal does not support rotation of secret.
Procedure to manually rotate the secret:
We require a re-deployment when the secret needs to be rotated.
Token acquisition
During this process, already connected clients will continue to work. But any new connection requests and renew/expire requests with old tokens can fail.
- expire all existing tokens
- rotate the secret by rolling upgrade, and
- generate new tokens
Token acquisition
Following steps describe how tokens can be acquired:
...
Token is stored in Zookeeper as properties in the path /tokenauthdelegation_token/tokens/<tokenUID>. S During server startup and token creation, scram credentials are generated during token creation. We will use password=HMAC, iterations=4096 for scram credential generation.and stored in memory (TokenCache).
Code Block | ||||
---|---|---|---|---|
| ||||
//Delegation Token Details for tokenID token123: Zookeeper persistence path /tokenauth/tokens/token123 { "version":1, "owner" : "owner", "renewer" : "renewer", "issueDateissueTimestamp" : "issueDateissueTimestamp", "tokenIDmaxTimestamp" : "sequence-numbermaxTimestamp", //Store SCRAM credentials as per KIP-84 "credentials": { "SCRAM-SHA-512"expiryTimestamp" : "expiryTimestamp", "tokenID" : "UUID"salt=djR5dXdtZGNqamVpeml6NGhiZmMwY3hrbg==,stored_key=sb5jkqStV9RwPVTGxG1ZJHxF89bqjsD1jT4S...==,server_key=...,iterations=4096", "SCRAM-SHA-256" : "salt=10ibs0z7xzlu6w5ns0n188sis5,stored_key=+Acl/wi1vLZ95Uqj8rRHVcSp6qrdfQIwZbaZBwM0yvo=,server_key=nN+fZauE6vG0hmFAEj/49+2yk0803y67WSXMYkgh77k=,iterations=4096" } }; |
SCRAM Extensions
,
}; |
SCRAM Extensions
SCRAM messages have an optional extensions field which is a comma-separated list of key=value pairs.
After KIP-84 implementation , an extension will be added to the first client SCRAM message to indicate
that authentication is SCRAM messages have an optional extensions field which is a comma-separated list of key=value pairs.
After KIP-84 implementation , an extension will be added to the first client SCRAM message to indicate
that authentication is being requested for a delegation token. This will enable Kafka broker to obtain
credentials and principal using a different code path for delegation tokens.
...
Code Block | ||||
---|---|---|---|---|
| ||||
public class DelegationTokenClient {
public TokenDetails generateToken(List<String> renewers, long maxLifeTime);
public boolean renewToken(bytes[] hmac, long renewPeriod);
public boolean expireToken(bytes[] hmac, long expireTimeStamp);
public boolean invalidateToken(bytes[] hmac);
public List<TokenDetails> describeTokens();
public void close();
} |
Command line tool
We will provide a CLI to acquire delegation tokens, renew tokens and to invalidate/expire tokens.
Changes to Java Clients (producer/consumer)
KIP-85 allows dynamic JAAS configuration for Kafka clients. After this we can easily configure the
delegation token for SCRAM-SHA-256 authentication.
Below diagram shows the steps required to use the delegation tokens.
...
hmac, long expireTimeStamp);
public boolean invalidateToken(bytes[] hmac);
public List<TokenDetails> describeTokens();
public void close();
} |
Command line tool
We will provide a commandline script to acquire delegation tokens, renew tokens, invalidate/expire and to describe tokens.
bin/kafka-delegation-token.sh --bootstrap-server broker1:9092 --create --renewer renewer1,renewer2 --max-life-time 1486750745585
bin/kafka-delegation-token.sh --bootstrap-server broker1:9092 --renew --hmac hmacString --renew-time-period 50745585
bin/kafka-delegation-token.sh --bootstrap-server broker1:9092 --expire --hmac hmacString --expiry-time-period 50745585
bin/kafka-delegation-token.sh --bootstrap-server broker1:9092 --describe --owner owner1,owner2
Changes to Java Clients (producer/consumer)
KIP-85 allows dynamic JAAS configuration for Kafka clients. After this we can easily configure the
delegation token for SCRAM-SHA-256 authentication.
Below diagram shows the steps required to use the delegation tokens.
ACL
Currently, we only allow a user to create delegation token for that user only. Renew and expire requests should come from owner or renewers of the token. So we don't need ACLs for create/renew/expire requests.
Owners and the renewers can always describe their own tokens. To describe others tokens, we can add DESCRIBE operation on Token Resource. In future, when we extend the support to allow a user to acquire delegation tokens for other users, then we can enable CREATE/DELETE operations.
Operation | Resource | API |
---|---|---|
DESCRIBE | Token | describeTokens |
CREATE | Token | createToken (Will be introduced in a future release) |
DELETE | Token | deleteToken (Will be introduced in a future release) |
Q/A
Q1. Is there any dependency on Hadoop APIs/Libraries?
...