Current state: "Accepted"
Discussion thread: here
Vote Thread: here
JIRA: KAFKA-7800
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Note: This KIP is based on KIP-339: Create a new IncrementalAlterConfigs API
Logging is a critical part of any system's infrastructure. It is the most direct way of observing what is happening with a system. In the case of issues, it helps us diagnose the problem quickly which in turn helps lower the MTTR.
Kafka supports application logging via the log4j library and outputs messages in various log levels (TRACE, DEBUG, INFO, WARN, ERROR). Log4j is a rich library that supports fine-grained logging configurations (e.g use INFO-level logging in kafka.server.ReplicaManager
and use DEBUG-level in kafka.server.KafkaApis
).
This is statically configurable through the log4j.properties file which gets read once at broker start-up.
A problem with this static configuration is that we cannot alter the log levels when a problem arises. It is severely impractical to edit a properties file and restart all brokers in order to gain visibility of a problem taking place in production.
It would be very useful if we support dynamically altering the log levels at runtime without needing to restart the Kafka process.
Log4j itself supports dynamically altering the log levels in a programmatic way and Kafka exposes a JMX API that lets you alter them. This allows users to change the log levels via a GUI (jconsole) or a CLI (jmxterm) that uses JMX.
There is one problem with changing log levels through JMX that we hope to address and that is Ease of Use:
Ideally, Kafka would support dynamically changing log levels and address all of the aforementioned concerns out of the box.
We propose extending the IncrementalAlterConfig/DescribeConfig Admin API with functionality for dynamically altering a single broker's log level.
This approach would also pave the way for even finer-grained logging logic (e.g log DEBUG level only for a certain topic) and would allow us to leverage the existing AlterConfigPolicy for custom user-defined validation of log-level changes.
These log-level changes will be temporary and reverted on broker restart - we will not persist them anywhere.
Users most likely need two operations for managing log levels - reading the currently-set log levels and altering them. Thus, we will add new functionality to the DescribeConfig and IncrementalAlterConfigs Admin APIs.
To differentiate between the normal Kafka config settings and the application's log level settings, we will introduce a new resource type - BROKER_LOGGERS
|
When resource_type=BROKER_LOGGER:
Cluster
resource (as used in IncrementalAlterConfigs/DescribeConfigs
operations).Let's define the log levels we support and the syslog severity level (as defined in RFC-5424) they correspond to:
TRACE - intended to enable TRACE logs - syslog level 7 and above |
To have these levels defined in code, we will create a new public class called LogLevelConfig, intended to be used by the AdminClient
package org.apache.kafka.common.config; /** * This class holds definitions for log level configurations related to Kafka's application logging. See KIP-412 for additional information */ public class LogLevelConfig { /* * NOTE: DO NOT CHANGE EITHER CONFIG NAMES AS THESE ARE PART OF THE PUBLIC API AND CHANGE WILL BREAK USER CODE. */ /** * The <code>FATAL</code> level designates a very severe error * that will lead the Kafka broker to abort. */ public static final String FATAL_LOG_LEVEL = "FATAL"; /** * The <code>ERROR</code> level designates error events that * might still allow the broker to continue running. */ public static final String ERROR_LOG_LEVEL = "ERROR"; /** * The <code>WARN</code> level designates potentially harmful situations. */ public static final String WARN_LOG_LEVEL = "WARN"; /** * The <code>INFO</code> level designates informational messages * that highlight normal Kafka events at a coarse-grained level */ public static final String INFO_LOG_LEVEL = "INFO"; /** * The <code>DEBUG</code> Level designates fine-grained * informational events that are most useful to debug Kafka */ public static final String DEBUG_LOG_LEVEL = "DEBUG"; /** * The <code>TRACE</code> Level designates finer-grained * informational events than the <code>DEBUG</code level. */ public static final String TRACE_LOG_LEVEL = "TRACE"; } |
We will add a new DYNAMIC_BROKER_LOGGER_CONFIG
type to the ConfigSource enum
|
We will change the behavior of two getter methods.
This is clearer and more user friendly. Previously, users would need to know to search for the ROOT logger's log level to figure out what log level their desired logger was logging at.
We will not be modifying the DescribeConfigs/IncrementalAlterConfigs request/response.
Let's go over the expected semantics when using them with the new resource type.
|
Request semantics (as defined in KIP-133) are conserved where applicable:
We will only support two out of the four operations for IncrementalAlterConfigs when the resource_type=BROKER_LOGGER
.
SET: Set the log level to the desired value
REMOVE: Unsets the log level of the logger. This effectively means it is set to the root logger's log level, as the logging library goes up the chain of configured loggers until it finds one. By default - the next logger in the hierarchy is root.
|
Request semantics (as defined in KIP-133 and KIP-339) are conserved where applicable:
In the case of an invalid config_value or an invalid/non-existent logger name, the broker will return an INVALID_CONFIG
(40) error for that config resource (BROKER_LOGGER).
INVALID_REQUEST will be returned when attempting to unset the ROOT logger's log level
kafka-configs.sh
will be extended to support the new resource type via --entity-type broker-logger
.
|
Since we are only adding new functionality under a new resource type, this KIP should not have compatibility issues with older versions.
Kafka will continue to expose the JMX API for configuring log levels. The only difference is that it will not return "null" for unset loggers now but rather return the root logger's log level.
Since we want to deprecate AlterConfigs
, that API will not support altering log levels.
We should be able to create a JUnit integration test inside AK that can call the Admin API methods to modify the log-level and have access to Log4j in order to verify that the levels are changed.
AlterConfigPolicy
implementations will need to be updated to account for the new config type.
config_type
field in Alter/Describe request/responses or new map of logger=>log_level