You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 14 Next »

Status

Current state: Under Discussion.

Discussion thread: https://www.mail-archive.com/dev@kafka.apache.org/msg99656.html

Vote thread:

JIRA:  Unable to render Jira issues macro, execution error.

Released: 

Pull request: 

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Kafka Connect does not provide an out-of-the-box facility to change log levels. When debugging connectors or the Connect framework, users have to update the log4j.properties file and restart the worker to see new logs. This is cumbersome in most cases, and restarting the worker sometimes hides bugs by resetting the internal state.

Proposed Changes

This change will introduce an /admin/loggers endpoint to a Connect worker. In addition, a new configuration property admin.listener will be provided to specify which listener to use for the admin endpoint. The endpoint will support the following operations:

Get a list of all loggers

$ curl -Ss http://localhost:8083/admin/loggers | jq
[
  ...
  "org.apache.kafka.connect.runtime.WorkerSinkTask",
  "org.apache.kafka.connect.runtime.WorkerSourceTask",
  ...
]

Get the log level of a specific logger

$ curl -Ss http://localhost:8083/admin/loggers/org.apache.kafka.connect.runtime.WorkerSinkTask | jq
{
   "level": "INFO"
}

Set the log level of a specific logger

$ curl -Ss -X PUT '{"level": "TRACE"}' http://localhost:8083/admin/loggers/org.apache.kafka.connect.runtime.WorkerSinkTask
// no output

Configuration Changes

An additional configuration property will be added to control where the admin endpoint will be made available.

Configuration NameDescriptionDefault BehaviorDomain
admin.listenersControl where the admin endpoint will be made available.the endpoint will be added to the rest of Connect's existing endpoints 

List of comma-separated URIs the REST API will listen on. An empty or blank string will disable this feature.

admin.listeners.https.Prefix for SSL settings when using HTTPS Look at Connect REST docs for a list of supported configs that  with this prefix.Look at Connect REST docs for a list of supported configs that can go with this prefix.

Example Usage

Example 1: Disable the admin endpoints

Set a blank string to the admin.listeners property to disable the entire /admin endpoint.

admin.listeners=

Example 2: Use a separate listener for /admin 

Bring up the admin endpoint on a separate listener:

admin.listeners=http://localhost:9093

Example 3: Use a separate listener for /admin (https)

Set up SSL properties in a separate https listener:

admin.listeners=https://localhost:9093
admin.listeners.https.ssl.truststore.location=/path/to/truststore.jks
admin.listeners.https.ssl.truststore.password=tpass
admin.listeners.https.ssl.keystore.location=/path/to/keystore.jks
admin.listeners.https.ssl.keystore.password=kpass

Compatibility, Deprecation, and Migration Plan

This is a new feature. Existing features will not be changed.

It must be noted that the changes proposed only work when log4j 1.x is used to log messages and is available on classpath. This is the current default in the AK distribution as well. Other loggers (logback, log4j2 etc) will not be supported and this feature (and others mentioned below in related work) will have to be reworked to support these frameworks.

Rejected Alternatives

  • Changing log levels in a single node of an application will affect other nodes in the cluster (for example, changing log level of a class in one Connect worker will update levels in all workers in a Connect cluster) and this new level will be persisted across node restarts. This is beyond the scope of this proposal.
  • Using JMX to change log levels is too cumbersome, difficult to secure and the other components are going to move away from it (look at KIP-412 where brokers are now using Admin Client to change log levels).

Related Work


  • No labels