Status

Current state: Under Discussion

Discussion thread: thread

JIRA: KAFKA-12399

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Kafka has been using log4j and providing a log4j appender also. KIP-653 upgrades log4j to log4j2 from its server-side modules, leaving log4j-appender and client-side modules from the upgrade.

This KIP is the following job of KIP-653; it proposes to provide a log4j2 equivalent of log4j-appender, with upgrading the omitted modules (i.e., tools and trogdor) into log4j2.

Before we proceed, here are several reasons why the Kafka community itself should provide log4j2 appender:

1. Dependency is best maintained when log4j2 and Kafka versions are updated together

Kafka core is dependent on log4j2; However, log4j2 Kafka appender is dependent on kafka client and in turn, kafka core. In short, they are dependent on each other.

If the Kafka community does not maintain the log4j2 Kafka appender, their versions will be updated independently, increasing the bug occurrence's complexity and potential. To minimize this problem, the Kafka community should provide a log4j2 appender as a submodule of Kafka (like they did in log4j-appender) and manage the dependency.

2. Absence of an affordable alternative

log4j2 community is providing Kafka appender by themselves, and there are also other open-source implementations. Although the maintainers are making a good effort on them, they have limitations:

1. Since their main focus is not Kafka itself, they provide only limited features available in Kafka Producer or log4j-appender. For example, Kafka supports very sophisticated security options, but only a few of them are supported by those appenders.
2. They are not transparently replaceable. For example, the Kafka appender provided by log4j2 community stores log message in Record key; in contrast, log4j-appender fixes key to null and stores log message in Record value. For this reason, tools assumpting log4j-appender are not compatible with those community implementations.

For the reasons above, migrating onto the log4j2 configuration is almost impossible as of the present. To resolve this problem, it is best to provide a successor of log4j-appender.

Public Interfaces

The users will be able to use org.apache.kafka:log4j2-appender:{kafka-version} dependency with log4j2 configuration syntax, like:

name=ExampleConfig
status=OFF
packages=org.apache.kafka.log4j2appender

appenders=kafkaAppender

appender.kafkaAppender.type=KafkaAppender
appender.kafkaAppender.name=kafkaAppender
appender.kafkaAppender.topic=test-topic
appender.kafkaAppender.brokerList=broker-0:9092,broker-1:9092,broker-2:9092
appender.kafkaAppender.layout.type=PatternLayout
appender.kafkaAppender.layout.pattern=[%d] %p %m (%c:%L)%n

rootLogger.level=INFO
rootLogger.appenderRefs=kafkaAppender
rootLogger.appenderRef.kafkaAppender.ref=kafkaAppender

Proposed Changes

The goal of this proposal is to provide a log4j2-equivalent of traditional log4j appender, following the principles below:

1. All configuration properties in log4j-appender must be supported in its log4j2 successor.
   • It should run by only converting legacy log4j syntax into log4j2.
   • It should provide a way to run legacy configuration for backward-compatibility.
2. It should have the same semantic as log4j-appender.
   • It should have the same format with log4j-appender for backward-compatibility. That is, the key must always be null, and the value must have a log message. The records produced by log4j-appender and its log4j2 successor should not be distinguishable.
3. It should reflect the recent changes made to the Producer.
   • If there is a change in Producer property, it should also be applied to the log4j2 appender.
   • If the name of the log4j-appender property and Producer property is different, it should support the latter one.

With log4j2-appender, the leaving modules (tools, trogdor) 's slf4j, log4j 1.x dependencies will be upgraded into log4j2 and additional log4j2 configuration file will be provided. For backward compatibility, these tools will use the log4j configuration file (tools-log4j2.properties) by default. But for informational purpose, the following message will be shown when user launches kafka-run-class.sh:

DEPRECATED: using log4j 1.x configuration. To use log4j 2.x configuration, run with: 'export KAFKA_LOG4J_OPTS="-Dlog4j.configurationFile=file:$base_dir/config/tools-log4j2.properties"'

As the message above states, the user can run the tools with log4j2 config file by setting `export KAFKA_LOG4J_OPTS="-Dlog4j.configurationFile={log4j2-config-file-path}"`. Thanks to log4j12-api, a compatibility bridge between log4j and log4j2, the tools can be run without any changes.

Compatibility, Deprecation, and Migration Plan

Since this module is a new one, there is no Deprecation or Migration Plan.

If a user hopes to use log4j2-appender with legacy log4j configuration, they can do it by:

1. Add a bridge dependency, org.apache.logging.log4j:log4j-1.2-api:{log4j2-version}, to the project.
2. Run java application with `-Dlog4j.configuration={log4j-config-path}` JVM parameter.

Rejected Alternatives

1. Leave this functionality to the community

In this approach, the users with log4j-appender are left with obsolete log4j configuration syntax without any affordable alternative. So rejected.