...

Vote Thread: here

JIRA: KAFKA-7800

Pull Request: PR#6903

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

...

Ideally, Kafka would support dynamically changing log levels and address all of the aforementioned concerns out of the box.

We propose extending the IncrementalAlterConfig/DescribeConfig Admin API with functionality for dynamically altering the a single broker's log level.

This approach would also pave the way for even finer-grained logging logic (e.g log DEBUG level only for a certain topic) and would allow us to leverage the existing AlterConfigPolicy for custom user-defined validation of log-level changes.
These log-level changes will be temporary and reverted on broker restart - we will not persist them anywhere.

...

Users most likely need two operations for managing log levels - reading the currently-set log levels and altering them. Thus, we will add new functionality to the DescribeConfig and IncrementalAlterConfigs Admin APIs.

Resource Type

To differentiate between the normal Kafka config settings and the application's log level settings, we will introduce a new resource type - BROKER_LOGGERS

When resource_type=BROKER_LOGGER:

• we will use the existing ACL for the Cluster resource (as used in IncrementalAlterConfigs/DescribeConfigs operations).
• we will only support one of eight values six values for the value of a config - the log levels (ALL, TRACE, DEBUG, INFO, WARN, ERROR, FATAL, OFF)

Log Level Definitions

Let's define the log levels we support and the syslog severity level (as defined in RFC-5424) they correspond to:

To have these levels defined in code, we will create a new public class called LogLevelConfig, intended to be used by the AdminClient

...

package org.apache.kafka.common.config;

/**
 * This class holds definitions for log level configurations related to Kafka's application logging. See KIP-412 for additional information
 */
public class LogLevelConfig {
    /*
     * NOTE: DO NOT CHANGE EITHER CONFIG NAMES AS THESE ARE PART OF THE PUBLIC API AND CHANGE WILL BREAK USER CODE.
     */

    /**
     * The <code>OFF<<code>FATAL</code> haslevel thedesignates highesta possiblevery ranksevere and iserror
     * intendedthat towill turnlead offthe logging.
Kafka broker    * WARNING: This is not recommended for useto abort.
     */
    public static final String OFFFATAL_LOG_LEVEL = "OFFFATAL";

    /**
     * The <code>FATAL<<code>ERROR</code> level designates aerror veryevents severe errorthat
     * thatmight willstill leadallow the Kafka broker to abortcontinue running.
     */
    public static final String FATALERROR_LOG_LEVEL = "FATALERROR";

    /**
     * The <code>ERROR<<code>WARN</code> level designates errorpotentially events that
     * might still allow the broker to continue runningharmful situations.
     */
    public static final String ERRORWARN_LOG_LEVEL = "ERRORWARN";

    /**
     * The <code>WARN<<code>INFO</code> level designates potentially harmful situations.informational messages
     */
    public static final String WARNthat highlight normal Kafka events at a coarse-grained level
     */
    public static final String INFO_LOG_LEVEL = "WARNINFO";

    /**
     * The <code>INFO<<code>DEBUG</code> levelLevel designates informational messagesfine-grained
     * informational events that are most thatuseful highlightto normaldebug Kafka events at a coarse-grained level
     */
    public static final String INFODEBUG_LOG_LEVEL = "INFODEBUG";

    /**
     * The <code>DEBUG<<code>TRACE</code> Level designates finefiner-grained
     * informational events thatthan arethe most useful to debug Kafka<code>DEBUG</code level.
     */
    public static final String DEBUGTRACE_LOG_LEVEL = "DEBUGTRACE";
}
    /**
     * The <code>TRACE</code> Level designates finer-grained
     * informational events than the <code>DEBUG</code level.
     */
    public static final String TRACE_LOG_LEVEL = "TRACE";

    /**
     * The <code>ALL</code> has the lowest possible rank and is intended to turn on all logging.
     */
    public static final String ALL_LOG_LEVEL = "ALL";
}

We will add a new DYNAMIC_BROKER_LOGGER_CONFIG type to the ConfigSource enum

public enum ConfigSource {

  DYNAMIC_BROKER_LOGGER_CONFIG, // dynamic broker logger config that is configured for a specific broker <--- NEW

Config Source

We will add a new DYNAMIC_BROKER_LOGGER_CONFIG type to the ConfigSource enum

Log4jController

We will change the behavior of two getter methods.

• Log4jController#getLogLevel()
  • It would previously return "Null log level." for a logger that did not have an explicitly-configured log level
  • It will now return the ROOT logger's log level when the log level is not explicitly set for a logger
• Log4jController#getLoggers()
  • It would return all the loggers and their associated log level in the format of "logger=level". When a logger did not have an explicitly-configured log level, it would return "null" as the log level
  • It will now return the ROOT logger's log level when the log level is not explicitly set for a logger

This is clearer and more user friendly. Previously, users would need to know to search for the ROOT logger's log level to figure out what log level their desired logger was logging at.

...

Request/Response Overview

We will not be modifying the DescribeConfigs/IncrementalAlterConfigs request/response.
Let's go over the expected semantics when using them with the new resource type.

DescribeConfigs

Request semantics (as defined in KIP-133) are conserved where applicable:

...

SET: Set the log level to the desired value
REMOVE: Sets the log level to NULL. In log4j, this means using the next higher-up subpackage logger (or root logger). Most classes have a log level of NULL if inspected through JMX: Unsets the log level of the logger. This effectively means it is set to the root logger's log level, as the logging library goes up the chain of configured loggers until it finds one. By default - the next logger in the hierarchy is root.

Request semantics (as defined in KIP-133 and KIP-339) are conserved where applicable:

...

In the case of an invalid config_value or an invalid/non-existent logger name, the broker will return an INVALID_CONFIG (40) error for that config resource (BROKER_LOGGER).
INVALID_REQUEST will be returned when attempting to unset the ROOT logger's log level

Tools Changes

kafka-configs.sh will be extended to support the new resource type via --entity-type broker-logger.

Examples:

Compatibility, Deprecation, and Migration Plan

...

Since we are only adding new functionality under a new resource type, this KIP should not have compatibility issues with older versions.

Kafka will continue to expose the JMX API for configuring log levelsKafka will continue to expose the JMX API for configuring log levels. The only difference is that it will not return "null" for unset loggers now but rather return the root logger's log level.

Since we want to deprecate AlterConfigs, that API will not support altering log levels.

...

1. Extend Kafka's JmxTool class to support changing log levels dynamically
   1. Does not properly address the concerns outlined in the Motivation section 
      1. Can abstract away a bit of the complexity but still has the fundamental flaws of no security and not being easy to use
2. Create new Admin API commands for reading and altering log levels
   1. Will result in more boilerplate and some code duplication.
   2. Will result in seemingly needless Admin API command bloat
   3. Altering log levels can be seen as a form of altering configurations and seems intuitive use the same API
   4. Will require separate Policy class
3. Add new config_type field in Alter/Describe request/responses or new map of logger=>log_level 
   1. Results in more code and field bloat in request/response without clear benefits of extendability
4. Support turning log levels to OFF
   1. Log4j supports a log level called OFF. We decided to not expose this as there is never a compelling reason to turn off application logs.