...
Kafka Connect does not provide an out-of-the-box facility to change log levels. When debugging connectors or the Connect framework, one has users have to update the log4j.properties
file and restart the worker to see new logs. This is cumbersome in most cases, and restarting the worker sometimes hides bugs by resetting the internal state.
Proposed Changes
The Kafka broker currently has a utility to adjust log levels via MBeans. This is a Scala utility, that we will rewrite in Java, move to kafka-clients package and use this utility to initialize JMX interface in the Kafka broker and Connect worker upon startup.
Public Interfaces
We propose adding the following log4j controller:
Code Block | ||
---|---|---|
| ||
public interface Log4jControllerMBean {
/**
* @return a list of all registered loggers
*/
List<String> getLoggers();
/**
* Get the effective log level for a given logger
*
* @param logger name of the logger
* @return its log level ("INFO", for example)
*/
String getLogLevel(String logger);
/**
* Set the log level for a logger
*
* @param logger name of the logger
* @param level desired level ("INFO", for example)
* @return true, if successfully set, false otherwise.
*/
boolean setLogLevel(String logger, String level);
} |
Applications will register this mBean with the mBean server via a LogLevelManager
utility class:
Code Block | ||
---|---|---|
| ||
public class LogLevelManager {
/**
* Create and register a JMX mBean called Log4jController in the specified domain
*
* @param domain domain of the mBean
*/
public static void registerLog4jController(String domain) {
// implementation
}
} |
Dynamic Log Levels in Connect
During startup, the Connect worker will use the LogLevelManager
utility to initialize dynamic logging as follows:
Code Block | ||
---|---|---|
| ||
LogLevelManager.registerLog4jController("kafka.connect"); |
This will provide a JMX bean Log4jController
in the kafka.connect
domain, that will include a Loggers
attribute, along with two operations: getLogLevel
and setLogLevel
that we can use to get or set log levels for individual loggers in the process.
Example Usage with JConsole
The following screenshots show to use JConsole to check and alter the log level of a class (The screenshots below show how to get and set the log levels of org.apache.kafka.connect.runtime.Worker).
...
...
...
Compatibility, Deprecation, and Migration Plan
The Kafka brokers already provide a similar feature. With this change, we bring about the following changes in it.
...
This change will introduce an /admin/loggers
endpoint to a Connect worker. In addition, a new configuration property admin.listener
will be provided to specify which listener to use for the admin endpoint. The endpoint will support the following operations:
Get a list of all loggers
Code Block | ||
---|---|---|
| ||
$ curl -Ss http://localhost:8083/admin/loggers | jq
[
...
"org.apache.kafka.connect.runtime.WorkerSinkTask",
"org.apache.kafka.connect.runtime.WorkerSourceTask",
...
] |
Get the log level of a specific logger
Code Block | ||
---|---|---|
| ||
$ curl -Ss http://localhost:8083/admin/loggers/org.apache.kafka.connect.runtime.WorkerSinkTask | jq
{
"level": "INFO"
} |
Set the log level of a specific logger
Code Block | ||
---|---|---|
| ||
$ curl -Ss -X PUT '{"level": "TRACE"}' http://localhost:8083/admin/loggers/org.apache.kafka.connect.runtime.WorkerSinkTask
// no output |
Configuration Changes
An additional configuration property will be added to control where the admin endpoint will be made available.
Configuration Name | Description | Default Behavior | Domain |
---|---|---|---|
admin.listeners | Control where the admin endpoint will be made available. | the endpoint will be added to the rest of Connect's existing endpoints | List of comma-separated URIs the REST API will listen on. An empty or blank string will disable this feature. |
admin.listeners.https. | Prefix for SSL settings when using HTTPS | Look at Connect REST docs for a list of supported configs that can go with this prefix. | Look at Connect REST docs for a list of supported configs that can go with this prefix. |
Example Usage
Example 1: Disable the admin endpoints
Set a blank string to the admin.listeners property to disable the entire /admin endpoint.
Code Block | ||
---|---|---|
| ||
admin.listeners= |
Example 2: Use a separate listener for /admin
Bring up the admin endpoint on a separate listener:
Code Block | ||
---|---|---|
| ||
admin.listeners="http://localhost:9093" |
Example 3: Use a separate listener for /admin (https)
Set up SSL properties in a separate https listener:
Code Block | ||
---|---|---|
| ||
admin.listeners="https://localhost:9093"
admin.listeners.https.ssl.truststore.location=/path/to/truststore.jks
admin.listeners.https.ssl.truststore.password=tpass
admin.listeners.https.ssl.keystore.location=/path/to/keystore.jks
admin.listeners.https.ssl.keystore.password=kpass |
Compatibility, Deprecation, and Migration Plan
This is a new feature. Existing features will not be changed
...
.
It must be noted that the changes proposed only work when log4j 1.x is used to log messages and is available on classpath. This is the current default in the AK distribution as well. Other loggers (logback, log4j2 etc) will not be supported and this feature (and others mentioned below in related work) will have to be reworked to support these frameworks.
...
- Changing log levels in a single node of an application will affect other nodes in the cluster (for example, changing log level of a class in one Connect worker will update levels in all workers in a Connect cluster) and this new level will be persisted across node restarts. This is beyond the scope of this proposal.Rest extensions were rejected, as Kafka doesn't offer a REST server and we are aiming for a consistent experience across Kafka and Connect in this proposal
- Using JMX to change log levels is too cumbersome, difficult to secure and the other components are going to move away from it (look at KIP-412 where brokers are now using Admin Client to change log levels).
Related Work
- KIP-412: Extend Admin API to support dynamic application log levels
- KIP-449: Add connector contexts to Connect worker logs
...