...
Current state: Approved for 3.1.0 (WIP)
Discussion thread: here
JIRA: KAFKA-13202
...
Because the HTTP call made to the OAuth/OIDC provider may time out or transiently fail, there will be a retry mechanism that waits between attempts. The number of attempts that are made (including the first attempt) are variable as it uses an exponential backoff approach; the first attempt to connect to the HTTP endpoint will be made immediately. If that first attempt fails, a second attempt will first wait a configurable number of milliseconds–loginRetryBackoffMs–
sasl.login.retry.backoff.ms–
before trying again. If that second attempt fails, the wait time (loginRetryBackoffMs
) will be doubled before a third attempt. This pattern repeats as needed up to the maximum of loginRetryMaxBackoffMs
wait time of of sasl.login.retry.backoff.max.ms
.
There are several configuration options for this callback handler. Sensitive configuration options and SASL extensions appear under the JAAS configuration (sasl.jaas.config
) while the rest are top-level configuration.
...
sasl.oauthbearer.token.endpoint.uriurl
: OAuth issuer token endpoint URIURLsasl.oauthbearer.scope.claim.name
: optional override name of the scope claim; defaults toscope
sasl.oauthbearer.sub.claim.name
: optional override name of the sub claim; defaults tosub
sasl.login.connect.timeout.ms
: optional value in milliseconds for HTTPS connect timeout; defaults to10000
sasl.login.read.timeout.ms
: optional value in milliseconds for HTTPS read timeout; defaults to10000
sasl.login.retry.backoff.ms
: optional value in milliseconds for the amount of time to wait between HTTPS call attempts; defaults to 100sasl.login.retry.backoff.max.backoff.ms
: optional value in milliseconds for the maximum wait for HTTPS call attempts (as described above); defaults to10000
...
sasl.login.callback.handler.class=...OAuthBearerLoginCallbackHandlersasl.login.connect.timeout.ms=15000
sasl.oauthbearer.token.endpoint.uriurl=https://myidp.example.com/oauth2/default/v1/token
sasl.jaas.config=...OAuthBearerLoginModule required \
clientId="abc123" \
clientSecret="S3cr3t!" \
scope="sales-pipeline" \
extension_organizationId="sales-emea" ;
...
In the above example, the OAuth provider’s sasl.oauthbearer.token.endpoint.uriurl
has been specified as well as an override of the default for sasl.login.connect.timeout.ms
. The values for clientId
and clientSecret
as provided by the OAuth provider for an “API” or “machine-to-machine” account are required in the JAAS configuration. The optional scope
value will allow the inclusion of a scope
parameter when requesting the token.
...
sasl.oauthbearer.jwks.endpoint.uriurl
: OAuth issuer's JWK Set endpoint URI from URL from which to retrieve the set of JWKs managed by the provider; this can be afile://
-based URL that points to a broker file system-accessible file-based copy of the JWKS data. This allows the JWKS data to be updated on the file system and refreshed on the broker when the file is updated, thus avoiding any HTTP(S) communication with the OAuth/OIDC providersasl.oauthbearer.jwks.endpoint.refresh.interval.ms
: optional value in milliseconds for how often to refresh the JWKS from the URL pointed to bysasl.oauthbearer.jwks.endpoint.uriurl
. Only used when using an HTTP(S)-based URI URL forsasl.oauthbearer.jwks.endpoint.uriurl
. Defaults to3600000
(1 hour)sasl.oauthbearer.
sub.claim.name: name of the scope from which to extract the subject claim from the JWT; defaults tosub
jwks.endpoint.retry.backoff.ms
: optional value in milliseconds for the amount of time to wait between HTTPS call attempts to retrieve the JWKS; only used when using an HTTP(S)-based URL forsasl.oauthbearer
.jwks.endpoint.url
; defaults to 100sasl.oauthbearer.
scope.claim.name: name of the scope from which to extract the scope claim from the JWT; defaults toscope
sasl.oauthbearer.clock.skew.seconds
: optional value in seconds for the clock skew between the OAuth/OIDC provider and the broker. Onlyjwks.endpoint.retry.backoff.max.ms
: optional value in milliseconds for the maximum wait for HTTPS call attempts to retrieve the JWKS; only used when using an HTTP(S)-based URI URL forsasl.oauthbearer
.jwks.endpoint.uri
. Defaults to30
url
; defaults to10000
expectedsasl.oauthbearer.
audiencesub.claim.
name
: name of the scope from which to extract the subject claim from the JWT; defaults tosub
sasl.oauthbearer.scope.claim.name
: name of the scope from which to extract the scope claim from the JWT; defaults toscope
sasl.oauthbearer.clock.skew.seconds
: optional value in seconds for the clock skew between the OAuth/OIDC provider and the broker. Only used when using an HTTP(S)-based URL forsasl.oauthbearer
.jwks.endpoint.url
. Defaults to30
sasl.oauthbearer.expected.audience
: The (optional) comma-delimited setting for the broker to use to verify that the JWT was issued for one of the expected audiences: The (optional) comma-delimited setting for the broker to use to verify that the JWT was issued for one of the expected audiences. The JWT will be inspected for the standard OAuthaud
claim and if this configuration option is set, the broker will match the value from JWT'saud
claim to see if there is an exact match. If there is no match, the broker will reject the JWT and authentication will fail.sasl.oauthbearer.expected.issuer
: Optional setting for the broker to use to verify that the JWT was created by the expected issuer. The JWT will be inspected for the standard OAuthiss
aud
claim and if this configuration option is set, the broker will match the value from JWT'siss
aud
claim to see if there is an exact match. If there is no match, the broker will reject the JWT and authentication will fail.
Here's an example of the configuration as a part of a Java properties file:
listener.name.<listener name>.oauthbearer.sasl.login.callback.handler.class=...OAuthBearerValidatorCallbackHandler
listener.name.<listener name>.oauthbearer.sasl.jaas.config=...OAuthBearerLoginModule required;
sasl.oauthbearer.jwks.endpoint.uri=https://myidp.example.com/oauth2/default/v1/keys
sasl.oauthbearer.scope.claim.name=scp
In the above configuration the broker points to the appropriate OAuth provider sasl.oauthbearer.jwks.endpoint.uri
to retrieve a the set of JWKs for validation. In this example, a non-default value for sasl.oauthbearer.scope.claim.name
has been provided because the provider uses scp
for the name of the scope claim in the JWT it produces.
...
sasl.oauthbearer.expected.issuer
: Optional setting for the broker to use to verify that the JWT was created by the expected issuer. The JWT will be inspected for the standard OAuthiss
claim and if this configuration option is set, the broker will match the value from JWT'siss
claim to see if there is an exact match. If there is no match, the broker will reject the JWT and authentication will fail.
Here's an example of the configuration as a part of a Java properties file:
listener.name.<listener name>.oauthbearer.sasl.server.callback.handler.class=...OAuthBearerValidatorCallbackHandler
listener.name.<listener name>.oauthbearer.sasl.jaas.config=...OAuthBearerLoginModule required;
sasl.oauthbearer.jwks.endpoint.url=https://myidp.example.com/oauth2/default/v1/keys
sasl.oauthbearer.scope.claim.name=scp
In the above configuration the broker points to the appropriate OAuth provider sasl.oauthbearer.jwks.endpoint.url
to retrieve a the set of JWKs for validation. In this example, a non-default value for sasl.oauthbearer.scope.claim.name
has been provided because the provider uses scp
for the name of the scope claim in the JWT it produces.
JWKS Management Logic
The JSON Web Key Set (JWKS) is a JSON document provided by the OAuth/OIDC provider that lists the keys used to sign the JWTs it issues.
Here is a sample JWKS JSON document:
{
"keys": [
{
"kty": "RSA",
"alg": "RS256",
"kid": "abc123",
"use": "sig",
"e": "AQAB",
"n": "..."
},
{
"kty": "RSA",
"alg": "RS256",
"kid": "def456",
"use": "sig",
"e": "AQAB",
"n": "..."
}
]
}
Without going into too much detail, the array of keys
enumerates the key data that the provider is using to sign the JWT signature. The key ID (kid
) is referenced by the JWT's header in order to match up the JWT's signing key with the key in the JWKS. During the validation step, the jose4j OAuth library will use the contents of the appropriate key in the JWKS to validate the signature.
Given that the JWKS is referenced by the JWT, the JWKS must be made available by the OAuth/OIDC provider so that a JWT can be validated.
The JWKS will be kept up-to-date in two main ways:
...
Warning |
---|
If the the URL or file that is specified cannot be read, the broker will fail to start up. In the case of an HTTP(S)-based URIURL, the configured configured It is also important that the JWKS is retrieved before the broker's ports are opened. Otherwise clients that connect to the broker before the JWKS is retrieved will experience spurious authentication failures (e.g. during broker restarts). |
...
- If the JWKS URL is HTTP(S)-based and if the broker hasn't already attempted to resolve the key ID, enqueue a background thread to reload the JWKS from the HTTP(S) endpoint. The broker will keep track of key ID resolution failures so it doesn't repeatedly attempt to do so. If the JWKS URL is
file://
-based, no remediatory remediation processing will occur. - Send an authentication failure to the client. Since it is unknown at this point in processing of the key ID is valid-but-missing or if the key ID is just invalid, the broker will always issue an authentication error. Between the authentication failure delay mechanism and any client retry, there may be sufficient time for the broker to update the JWKS.
...
./bin/kafka-run-class.sh org.apache.kafka.tools.OAuthCompatibilityTool \
--client-id foo \
--client-secret bar \
--token-endpoint-uriurl https://example.com/oauth2/v1/token \
--jwks-endpoint-uriurl https://example.com/oauth2/v1/keys
...