This page is meant as a template for writing a KIP. To create a KIP choose Tools->Copy on this page and modify with your content and replace the heading with the next KIP number and a description of your issue. Replace anything in italics with your own description.
Current state: "Under Discussion"
Discussion thread: here
JIRA: KAFKA-7800
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Note: This KIP is based on KIP-339: Create a new IncrementalAlterConfigs API
Logging is a critical part of any system's infrastructure. It is the most direct way of observing what is happening with a system. In the case of issues, it helps us diagnose the problem quickly which in turn helps lower the MTTR.
Kafka supports application logging via the log4j library and outputs messages in various log levels (TRACE, DEBUG, INFO, WARN, ERROR). Log4j is a rich library that supports fine-grained logging configurations (e.g use INFO-level logging in kafka.server.ReplicaManager
and use DEBUG-level in kafka.server.KafkaApis
).
This is statically configurable through the log4j.properties file which gets read once at broker start-up.
A problem with this static configuration is that we cannot alter the log levels when a problem arises. It is severely impractical to edit a properties file and restart all brokers in order to gain visibility of a problem taking place in production.
It would be very useful if we support dynamically altering the log levels at runtime without needing to restart the Kafka process.
Log4j itself supports dynamically altering the log levels in a programmatic way and Kafka exposes a JMX API that lets you alter them. This allows users to change the log levels via a GUI (jconsole) or a CLI (jmxterm) that uses JMX.
There is one problem with changing log levels through JMX that we hope to address and that is Ease of Use:
Ideally, Kafka would support dynamically changing log levels and address all of the aforementioned concerns out of the box.
We propose extending the IncrementalAlterConfig/DescribeConfig Admin API with functionality for dynamically altering the broker's log level.
This approach would also pave the way for even finer-grained logging logic (e.g log DEBUG level only for a certain topic) and would allow us to leverage the existing AlterConfigPolicy for custom user-defined validation of log-level changes.
These log-level changes will be temporary and reverted on broker restart - we will not persist them anywhere.
Users most likely need two operations for managing log levels - reading the currently-set log levels and altering them. Thus, we will add new functionality to the DescribeConfig and IncrementalAlterConfigs Admin APIs.
To differentiate between the normal Kafka config settings and the application's log level settings, we will introduce a new resource type - BROKER_LOGGERS
|
When resource_type=BROKER_LOGGER:
Cluster
resource (as used in IncrementalAlterConfigs/DescribeConfigs
operations).We will not be modifying the DescribeConfigs/IncrementalAlterConfigs request/response.
Let's go over the expected semantics when using them with the new resource type.
|
Request semantics (as defined in KIP-133) are conserved where applicable:
We will only support two out of the four operations for IncrementalAlterConfigs when the resource_type=BROKER_LOGGER
.
SET: Set the log level to the desired value
REMOVE: Sets the log level to NULL
. In log4j, this means using the next higher-up subpackage logger (or root logger). Most classes have a log level of NULL
if inspected through JMX.
|
Request semantics (as defined in KIP-133 and KIP-339) are conserved where applicable:
In the case of an invalid config_value or an invalid/non-existent logger name, the broker will return an INVALID_CONFIG
(40) error for that config.
kafka-configs.sh
will be extended to support the new resource type via --entity-type broker-logger
.
|
Since we are only adding new functionality under a new resource type, this KIP should not have compatibility issues with older versions.
Kafka will continue to expose the JMX API for configuring log levels.
Since we want to deprecate AlterConfigs
, that API will not support altering log levels.
We should be able to create a JUnit integration test inside AK that can call the Admin API methods to modify the log-level and have access to Log4j in order to verify that the levels are changed.
AlterConfigPolicy
implementations will need to be updated to account for the new config type.
config_type
field in Alter/Describe request/responses or new map of logger=>log_level