Versions Compared

Old Version 57

changes.mady.by.user Manikumar Reddy O.

Saved on

compared with

New Version 58

changes.mady.by.user Manikumar Reddy O.

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Current state: ["DISCUSSION"]. 

Discussion thread: here

JIRA: KAFKA-1696 

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

...

Field

Description

Owner

Kakfa Principal which requested the delegation token

IssueDate

timestamp (in msec) when this token was generated generated. Unit is milliseconds since beginning of the epoch (midnight Jan 1, 1970 (UTC)).

ExpiryDate

timestamp (in msec) at which this token expires expires. Unit is milliseconds since beginning of the epoch (midnight Jan 1, 1970 (UTC)).

TokenId

Sequence number/UUID to ensure uniqueness 

HMAC
Keyed-hash message authentication code
Renewer
Renewers list

...

Code Block
RenewDelegationTokenRequest => HMAC  renewPeriod
  HMAC => bytes
  renewPeriod => INT64
renewPeriod in milli seconds

Field

Description

HMAC

HMAC of the delegation token to be renewed

renewPeriod

 
RenewDelegationTokenResponse

...

  1. delegation.token.max.lifetime.sec : The token has a maximum lifetime beyond which it cannot be renewed any more. Default value 7 days.
  2. delegation.token.expiry.time.sec : The token validity time in seconds before the token needs to be renewed. Default value 1 day.
  3. delegation.token.master.key : Secret/masterKey to generate and verify delegation tokens. This masterKey needs to be configured with all the brokers.

Proposed Changes

Token

The Kafka authentication token is modeled after the Hadoop user delegation token. The token will consist of:

 

TokenID:

  • Owner ID -- Username that this token will authenticate as
  • Renewers ID -- designated renewers list
  • Issue date -- timestamp (in msec) when this token was generated
  • Expiry date -- timestamp (in msec) at which this token expires
  • TokenUID -- Sequence number/UUID to ensure uniqueness

TokenAuthenticator(HMAC) := HMAC_SHA1(master key, TokenUID)

Authentication Token := (TokenID, TokenAuthenticator(HMAC))

Master Secret Key

The secret is used to generate and verify delegation tokens. This is supplied using config option. This masterKey needs to be configured with all the brokers. The current proposal does not support rotation of masterKey. We a requires a re-deployment when the masterKey needs to be rotated.

 Token acquisition

Following steps describe how tokens can be acquired:

  • A  (Admin/Delegation TokenDelegationToken) client connects with one of the kafka broker. Client must be authenticated using any of the available secure channels so it must have a way to authenticate, i.e. Kerberos keytab or TGT.

  • Once a client is authenticated, it will make a broker side call to issue a delegation token.  The request for delegation token will have to contain an optional renewer identity and max lifetime for token. The renewer is the user that is allowed to renew this token before the max lifetime expires. Renewer will default to the owner if not provided and Max life time will default to a server side config value (default  days) Brokers will allow a token to be renewed until maxLifeTime but a token will still expire if not renewed by the expiry time. The expiry time will be a broker side configuration and will default to min (24 hours, maxlifeTime) . A Delegation Token request can be represented as class DelegationTokenRequest(renewer: Set[KafkaPrincipal], maxLifeTime: long). The owner is implicit in the request connection as the user who requested the delegation token.

  • The broker generates a shared secret based on HMAC-SASM(a Password/Secret shared between all brokers, randomly generated tokenId). We can represent a token as scala case class DelegationToken(owner: KafkaPrincipal, renewer: Set[KafkaPrincipal], maxLifeTime: long, id: String, hmac: String, expirationTime: long)

  • Broker stores this token in its in memory cache. Broker also stores the DelegationToken without the hmac in the zookeeper. As all brokers share the Password/Secret to generate the HMAC-SASM, they can read the request info from zookeeper , generate the hmac and store the delegation token in local cache.

  • All brokers will have a cache backed by zookeeper so they will all get notified whenever a new token is generated and they will update their local cache whenever token state changes.

  • Broker returns the token to Client. Client is expected to only make delegation token request over an encrypted channel so the token in encrypted over the wire.

  • Client is free to distribute this token to other Kafka clients (Producer/Consumers). It is the client’s responsibility to distribute the token securely.

...

  • SCRAM is a suitable mechanism for authentication using delegation tokens. KIP-84 proposes to support  SASL SCRAM mechanisms. Kafka clients can authenticate using SCRAM-SHA-256, providing the delegation token HMAC as password.

  • Server will look up the token from its token cache, if it finds a match and token is not expired it will authenticate the client and the identity will be established as the owner of the delegation token.

  • If the token is not matched or token is expired, broker throws appropriate exception back and does not allow the client to continue.

...

 In case of a password compromise scenario all the tokens can be deleted from zookeeper and this will result in all the tokens to be invalidated. We can provide a simple CLI tool for this. 

Secret/Master Key

Secret/masterKey is used to generate and verify delegation tokens. This is supplied using config option. This masterKey needs to be configured with all the brokers. The current proposal does not support rotation of masterKey. We a requires a re-deployment when the masterKey needs to be rotated.

Token Details in Zookeeper

...