THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
| Warning |
|---|
If the the URL or file that is specified cannot be read, the broker will fail to start up. In the case of an HTTP(S)-based URI, the configured It is also important that the JWKS is retrieved before the broker's ports are opened. Otherwise clients that connect to the broker before the JWKS is retrieved will experience spurious authentication failures (e.g. during broker restarts). |
It is expected that an OAuth provider will publish a new JWKS well in advance of issuing any JWTs that contain those keys. However, in the case that a broker receives a JWT with a key ID that it doesn't have stored in its cached JWKS, the broker will need to remediate the issue:
- If the JWKS URL is HTTP(S)-based and if the broker hasn't already attempted to resolve the key ID, enqueue a background thread to reload the JWKS from the HTTP(S) endpoint. The broker will keep track of key ID resolution failures so it doesn't repeatedly attempt to do so. If the JWKS URL is
file://-based, no remediatory processing will occur. - Send an authentication failure to the client. Since it is unknown at this point in processing of the key ID is valid-but-missing or if the key ID is just invalid, the broker will always issue an authentication error. Between the authentication failure delay mechanism and any client retry, there may be sufficient time for the broker to update the JWKS.
The common case will be that the key ID is one that has been published and accessed recently. The broker shouldn’t need to reach out to the JWKS endpoint in the on-demand fashion described above normally.
| Warning |
|---|
The broker will not perform any loading of the JWKS in the network thread. This needs to be performed either prior to opening the ports during startup (as described above) or performed in a background thread. |
The code will need to have a means to expunge old JWKS that are no longer needed. If there is a tangible benefit, perhaps the broker can keep track of recently removed/aged out key IDs so as to provide a more helpful message to the user.
Broker-to-broker Support
The use of OAuth credentials for broker-to-broker communication will continue to be supported. As with the existing implementation, users can specify the protocols and implementations to use for broker-based communication. This would require providing the appropriate configuration for both client login and broker validation.
...
In addition to unit and integration tests, there will be a standalone tool in the tools directory/module : named org.apache.kafka.tools.OAuthCompatibilityTest. This test can be run via the existing bin/kafka-run-class.sh script thusly:
KAFKA_OPTS=""
export KAFKA_OPTS="$KAFKA_OPTS -DclientId=$client_id"
export KAFKA_OPTS="$KAFKA_OPTS -DclientSecret=$client_secret"
export KAFKA_OPTS="$KAFKA_OPTS -Dscope=api://$client_id/.default"
export KAFKA_OPTS="$KAFKA_OPTS -DtokenEndpointUri=$token_endpoint_uri"
export KAFKA_OPTS="$KAFKA_OPTS -DjwksEndpointUri=$jwks_uri"
export KAFKA_OPTS="$KAFKA_OPTS -DclockSkew=30"
export KAFKA_OPTS="$KAFKA_OPTS -DexpectedAudience=$expected_audience"
./bin/kafka-run-class.sh org.apache.kafka.tools.OAuthCompatibilityTest./bin/kafka-run-class.sh org.apache.kafka.tools.OAuthCompatibilityTest \
--client-id foo \
--client-secret bar \
--token-endpoint-uri https://example.com/oauth2/v1/token \
--jwks-endpoint-uri https://example.com/oauth2/v1/keys
As seen in the example invocation above, the various Kafka configuration options for the client and server broker properties are passed in via the existing KAFKA_OPTS environment variablecommand line options. The test will connect to remote systems as needed to authenticate, retrieve tokens, retrieve JWKS, and perform validation (nothing is "mocked"). The command line options can be listed using the standard --help command line option.
This tool serves three basic purposes:
...
Additionally debugging can be selectively enabled using the standard tools-log4j.properties file if errors are detected by the tool. All configuration options supported by the client and server are also supported by the tool; it attempts to be as faithful as possible to the runtime logic to minimize discrepancies between the tool and the client and serverbroker.
Compatibility, Deprecation, and Migration Plan
...