...
Ideally, Kafka would support dynamically changing log levels and address all of the aforementioned concerns out of the box.
We propose extending the IncrementalAlterConfig/DescribeConfig Admin API with functionality for dynamically altering the a single broker's log level.
This approach would also pave the way for even finer-grained logging logic (e.g log DEBUG level only for a certain topic) and would allow us to leverage the existing AlterConfigPolicy for custom user-defined validation of log-level changes.
These log-level changes will be temporary and reverted on broker restart - we will not persist them anywhere.
...
DescribeConfigs Request (Version: 2) => [resource [config_name]] include_synonyms
resource => resource_type resource_name
resource_type => INT8 # BROKER_LOGGER (8)
resource_name => STRING # ID of broker
config_name => STRING
include_synonyms => BOOLEAN # ignored
DescribeConfigs Response (Version: 2) => throttle_time_ms entities
throttle_time_ms => INT32
entities => error_code error_message resource configs
error_code => INT16
error_message => STRING
resource => resource_type resource_name
resource_type => INT8 # BROKER_LOGGER (8)
resource_name => STRING # ID of broker
configs => [config_entry synonym]
config_entry =>
config_name => STRING # logger name - e.g kafka.server.ReplicaManager
config_value => STRING # log level - e.g INFO. Returns the root logger's log level if this logger is not set explicitly yet
read_only => BOOLEAN # false always
config_source => INT8 # (DYNAMIC_BROKER_LOGGER_CONFIG)
is_sensitive => BOOLEAN # false always
synonym => # empty always
config_name => STRING
config_value => NULLABLE_STRING
config_source => INT8
|
...
SET: Set the log level to the desired value
REMOVE: Sets Unsets the log level to the same as of the logger. This effectively means it is set to the root logger's log level, as the logging library goes up the chain of configured loggers until it finds one. By default - the next logger in the hierarchy is root.
IncrementalAlterConfigsOp => INT8
0: SET
1: REMOVE # sets log level to the root logger's level
2: APPEND # NOT SUPPORTED
3: SUBTRACT # NOT SUPPORTED
IncrementalAlterConfigsRequest (Version: 0) => [resources] validate_only
validate_only => BOOLEAN
resources => resource_type resource_name [configs]
resource_type => INT8 # BROKER_LOGGER (8)
resource_name => STRING # ID of broker
configs => config_name config_op config_value
config_name => STRING
config_op => INT8 # support SET and APPEND only
config_value => NULLABLE_STRING
IncrementalAlterConfigsResponse (Version: 0) => [responses]
responses => resource_type resource_name error_code error_message
resource_type => INT8 # BROKER_LOGGER (8)
resource_name => STRING # ID of broker
error_code => INT16
error_message => NULLABLE_STRING
|
...
In the case of an invalid config_value or an invalid/non-existent logger name, the broker will return an INVALID_CONFIG (40) error for that config resource (BROKER_LOGGER).
INVALID_REQUEST will be returned when attempting to unset the ROOT logger's log level
Tools Changes
kafka-configs.sh will be extended to support the new resource type via --entity-type broker-logger.
...
bin/kafka-configs.sh --bootstrap-server localhost:9092 --describe --entity-type broker-logger loggers --entity-name 0
bin/kafka-configs.sh --bootstrap-server localhost:9092 --alter --add-config "kafka.server.ReplicaManager=WARN,kafka.server.KafkaApis=DEBUG" --entity-type broker-logger loggers --entity-name 0
bin/kafka-configs.sh --bootstrap-server localhost:9092 --alter --delete-config kafka.server.ReplicaManager --entity-type broker-logger loggers --entity-name 0
|
...