...

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Kafka has been using log4j and providing a log4j appender also. KIP-653 upgrades log4j to log4j2 from its server-side modules, leaving , but some modules (log4j-appender and client-side modules from the upgrade.

This KIP is the following job of KIP-653; it proposes to provide a log4j2 equivalent of log4j-appender, with upgrading the omitted modules (i.e., tools and trogdor) into log4j2.

Before we proceed, here are several reasons why the Kafka community itself should provide log4j2 appender:

1. Dependency is best maintained when log4j2 and Kafka versions are updated together

Kafka core is dependent on log4j2; However, log4j2 Kafka appender is dependent on kafka client and in turn, kafka core. In short, they are dependent on each other.

If the Kafka community does not maintain the log4j2 Kafka appender, their versions will be updated independently, increasing the bug occurrence's complexity and potential. To minimize this problem, the Kafka community should provide a log4j2 appender as a submodule of Kafka (like they did in log4j-appender) and manage the dependency.

2. Absence of an affordable alternative

log4j2 community is providing Kafka appender by themselves, and there are also other open-source implementations. Although the maintainers are making a good effort on them, they have limitations:

1. Since their main focus is not Kafka itself, they provide only limited features available in Kafka Producer or log4j-appender. For example, Kafka supports very sophisticated security options, but only a few of them are supported by those appenders.
2. They are not transparently replaceable. For example, the Kafka appender provided by log4j2 community stores log message in Record key; in contrast, log4j-appender fixes key to null and stores log message in Record value. For this reason, tools assumpting log4j-appender are not compatible with those community implementations.

, tools, and trogdor) were omitted, since providing log4j2 appender was beyond the scope of the proposal. As of the KIP-653 was passed (before 2.8.0), trogdor was not separated from tools yet, and tools depended upon log4j-appender, since VerifiableLog4jAppender uses it. It is why those three modules were excluded from the upgrade, and both log4j and log4j2 artifacts came to co-exist after the upgrade.

However, as the preview build is released and tested, this approach turned out to be wrong: some users reported that when running Kafka with log4j configuration file (which is allowed for backward compatibility), the logging message is sometimes not appended properly to the logger file. This problem was resolved after removing the log4j artifacts from the classpath.

This proposal aims to entirely remove the log4j artifact from the classpath by providing a log4j2 equivalent of log4j-appender, and upgrading all the omitted modules (i.e., tools and trogdor) into log4j2 to avoid the problem above. Plus, it can also allow the users of the deprecated log4j-appender to upgrade. For the reasons above, migrating onto the log4j2 configuration is almost impossible as of the present. To resolve this problem, it is best to provide a successor of log4j-appender.

Public Interfaces

The users will be able to use org.apache.kafka:log4j2-appender:{kafka-version} dependency with log4j2 configuration syntax, like:

...

1. Add a bridge dependency, org.apache.logging.log4j:log4j-1.2-api:{log4j2-version}, to the project.
2. Run java application with `-Dlog4j.configuration={log4j-config-path}` JVM parameter.

Rejected Alternatives

Leave the omitted modules (tools, trogdor) to use log4j 1

...

.x.

This approach has the following problems:

1. Inconsistency: if a user wants to change the logging of tools or trogdor, they have to deal with the deprecated log4j configuration syntax; KIP-653 still supports to use of log4j configuration for backward compatibility, but it will be removed in the future.
2. Complexity: it makes the classpath generation logic in shell scripts complex.

Instead of providing log4j2-appender, make use of the already-existing log4j2 appender provided by the community.

As far as I know, all of the major community appenders have a different format with log4j-appender; For example, the Kafka appender provided by log4j2 community stores log message in the Record key; in contrast, log4j-appender fixes the key to null and stores log message in the Record value. For this reason, the projects depending on log4j-appender are not compatible with those community implementations, not transparently replaceable.

Remove VerifiableLog4jAppender from tools, and upgrade the omitted modules (tools, trogdor) to use log4j2.

To remove any existing tool, regardless of whether it is used frequently or not, we have to deprecate it first and remove it after a reasonable period of time. This approach will make the adoption of log4j2 too lateIn this approach, the users with log4j-appender are left with obsolete log4j configuration syntax without any affordable alternative. So rejected.