Status

Current state: Under Discussion

...

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Kafka has been using log4j and providing a log4j appender also. KIP-653upgrades log4j to log4j2, but some modules (log4j-appenderi.e., tools, and trogdor, shell) were omitted, since providing log4j2 appender was beyond the scope of the proposal. As of the they depend on log4j-appender - as of the KIP-653 was passed (before 2.8.0), trogdor was and shell were not separated from tools yet, and tools depended upon log4j-appender, since VerifiableLog4jAppender uses it. It is why those three modules were excluded from the upgrade, and both For this reason, both of log4j and log4j2 artifacts came to co-exist in the classpath after the upgrade.

However, as the preview build is released and tested, this approach turned out to be wrong: problematic; some users reported that when running Kafka with log4j configuration file (which is allowed for backward compatibility), the logging message is sometimes not appended properly to the logger file. This problem was resolved after by removing the log4j artifacts from the classpath.

This proposal aims to entirely remove the log4j artifact from the classpath by providing a log4j2 equivalent of log4jdeprecate log4j-appender , and upgrading upgrade all the omitted modules (i.e., tools, trogdor, and trogdorshell) into log4j2 to entirely remove the log4j artifact from the classpath and avoid the problem above. Plus, it can also allow the users of the deprecated log4j-appender to upgrade. 

Public Interfaces

The users will be able to use org.apache.kafka:log4j2-appender:{kafka-version} dependency with log4j2 configuration syntax, like:

name=ExampleConfig
status=OFF
packages=org.apache.kafka.log4j2appender

appenders=kafkaAppender

appender.kafkaAppender.type=KafkaAppender
appender.kafkaAppender.name=kafkaAppender
appender.kafkaAppender.topic=test-topic
appender.kafkaAppender.brokerList=broker-0:9092,broker-1:9092,broker-2:9092
appender.kafkaAppender.layout.type=PatternLayout
appender.kafkaAppender.layout.pattern=[%d] %p %m (%c:%L)%n

rootLogger.level=INFO
rootLogger.appenderRefs=kafkaAppender
rootLogger.appenderRef.kafkaAppender.ref=kafkaAppender

Proposed Changes

The goal of this proposal is to provide a log4j2-equivalent of traditional log4j appender, following the principles below:

1. All configuration properties in log4j-appender must be supported in its log4j2 successor.
   • It should run by only converting legacy log4j syntax into log4j2.
   • It should provide a way to run legacy configuration for backward-compatibility.
2. It should have the same semantic as log4j-appender.
   • It should have the same format with log4j-appender for backward-compatibility. That is, the key must always be null, and the value must have a log message. The records produced by log4j-appender and its log4j2 successor should not be distinguishable.
3. It should reflect the recent changes made to the Producer.
   • If there is a change in Producer property, it should also be applied to the log4j2 appender.
   • If the name of the log4j-appender property and Producer property is different, it should support the latter one.

tools, trogdor, and shell module will use log4j2 instead of log4j, like other modules do in KIP-653.

Proposed Changes

1. Deprecate log4j-appender

2. Remove dependency on log4j-appender from the project

The tools using log4j-appender (e.g., VerifiableLog4jAppender) will be updated to use log4j2 Kafka appender instead. (We can transparently replace the dependencies since log4j2 Kafka appender uses the same format as log4j-appender.) That is, log4j-appender will be published as a part of the official release but not used by the other subprojects anymore.

The leaving modules (tools, trogdor, shell) 's With log4j2-appender, the leaving modules (tools, trogdor) 's slf4j, log4j 1.x dependencies will be upgraded into log4j2 and additional log4j2 configuration file will be provided. For backward compatibility, these tools will use the log4j configuration file (tools-log4j2.properties) by default. But for informational purpose, the following message will be shown when user launches kafka-run-class.sh:

...

As the message above states, the user can run the tools scripts with log4j2 config file by setting `export KAFKA_LOG4J_OPTS="-Dlog4j.configurationFile={log4j2-config-file-path}"`. Thanks to log4j12-api, a compatibility bridge between log4j and log4j2, the tools scripts can be run without any changes.

3. Provide a migration guide

We will provide a detailed migration guide to log4j2 Kafka appender for the users of the log4j-appender.

Compatibility, Deprecation, and Migration Plan

Since this module is a new one, there is no Deprecation or Migration Plan.

If a user hopes to use log4j2-appender with legacy log4j configuration, they can do it by:

1. Add a bridge dependency, org.apache.logging.log4j:log4j-1.2-api:{log4j2-version}, to the project.
2. Run java application with `-Dlog4j.configuration={log4j-config-path}` JVM parameter.

log4j-appender will be updated and published as new version releases to a certain point (perhaps, Apache Kafka 4.0) and entirely be removed from the project then. But the exact point is totally up to the community's decision.

Rejected Alternatives

Leave the omitted modules (tools, trogdor) to use log4j 1.x.

This approach has the following problems:

1. Inconsistency: if a user wants to change the logging of tools or trogdor, they have to deal with the deprecated log4j configuration syntax; KIP-653 still supports to use of log4j configuration for backward compatibility, but it will be removed in the future.
2. Complexity: it makes the classpath generation logic in shell scripts complex.

...

Provide Kafka's own log4j2-appender

...

which succeeds log4j-appender.

It will be almost identical to the log4j2's Kafka appender; It isn’t technically worthy.As far as I know, all of the major community appenders have a different format with log4j-appender; For example, the Kafka appender provided by log4j2 community stores log message in the Record key; in contrast, log4j-appender fixes the key to null and stores log message in the Record value. For this reason, the projects depending on log4j-appender are not compatible with those community implementations, not transparently replaceable.

Remove VerifiableLog4jAppender from tools, and upgrade the omitted modules (tools, trogdor) to use log4j2.

To remove any existing tool, regardless of whether it is used frequently or not, we have to deprecate it first and remove it after a reasonable period of time. This approach will make the adoption of log4j2 too late.

Final Notes

As of log4j2 2.16.0, log4j2's Kafka appender ignores logging messages from 'org.apache.kafka' to prevent recursive logging. This issue is now working on progress in