Status

Current state: Under Discussion Accepted

Discussion thread: thread

JIRA: KAFKA-9366

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

...

The problem caused by the obsolete log4j version is not limited to security problems like CVE-2019-17571. Most users are now familiar with log4j2 configurations(2.x) syntax, not log4j. But 1.x. For this reason, when they want are trying to customize the logging configuration of Apache Kafka , or Kafka Connect, and Kafka Streams application, they have to work with outdated, dismissed old configuration format.

This KIP proposes to upgrade the log4j 1.x dependencies into log4j2 from the Server-side of Kafka. (For the exact definition of 'server-side', please refer to the 'Which modules will be influenced?' subsection.)

Public Interfaces

This KIP proposes the following:

1. Replace log4j 1.x dependency into 2.x, with converting all 1.x API calls with 2.x equivalents.
2. Provide a way to use log4j2 configuration, with backward compatibility to log4j.

Proposed Changes

1. server-side dependency from log4j into log4j2, along with their slf4j bindings.
2. User-interfacing configurations (like broker logging config), provide additional log4j2-equivalent configuration with backward compatibility.
3. For non-user interfacing configurations (like test config), all of them will be migrated into log4j2.

Proposed Changes

0. Which modules will be influenced?

The following modules will be updated:

• clients: core, metadata, raft, server-common, and storage modules are directly dependent on clients module. So, We should include it.
• connect
• core
• metadata
• raft
• storage
• streams: this module directly depends on clients.
• embedded zookeeper

The following modules are not the scope of this proposal with some reasons:

• log4j-appender: This module should not be touched for the users, and its log4j2 equivalent should be provided independently. However, it is above the scope of this proposal.
• tools: VerifiableLog4jAppender depends on log4j-appender. So, we can't migrate them until log4j2-appender is ready.
• trogdor: As of this KIP was passed, trogdor was a part of tools. So, it was excluded.

1. clients

slf4j, log4j dependencies (org.slf4j:slf4j-log4j12, log4j:log4j) will be upgraded into log4j2 (org.apache.logging.log4j:log4j-slf4j-impl, org.apache.logging.log4j). The test logging configuration (src/test/resources/log4j.properties) will be migrated into log4j2. (In this case, we don't care about the backward-compatibility.)

2. connect

slf4j, log4j 1.x dependencies will be upgraded into log4j2 and additional log4j2 configuration file will be provided.

...

For backward compatibility, Kafka broker will use the log4j configuration file file (connect-log4j2.properties) by default. But for informational purpose, the following message will be shown in default modewhen user launches connect-standalone.sh, connect-mirror-maker.sh, and connect-distributed.sh:

DEPRECATED: using log4j 1.x configuration. To use log4j 2.x configuration, run with: 'export KAFKA_LOG4J_OPTS=\"-Dlog4j.configurationFile=file:$base_dir/../config/connect-log4j2.properties\"'

As the message above states, the user can run Kafka broker with log4j2 config file by setting `export KAFKA_LOG4J_OPTS="-Dlog4j.configurationFile={log4j2-config-file-path}"`. Thanks to log4j12-api, a compatibility bridge between log4j and log4j2, Kafka broker can be run without any changes. Since a log4j2 equivalent for traditional built-in log4j config (log4j2.properties) will be provided, the user can make use of it if they want.

The The test logging configuration (src/test/resources/log4j.properties) will be updated into log4j2. In this case, we don't care about the backward-compatibility.

2. Client

The test logging configuration (src/test/resources/log4j.properties) will be updated into log4j2.

3. Connect

3. core

Like connect, slf4j, log4j 1.x dependencies will be upgraded into log4j2 and additional log4j2 configuration file will be provided.

For backward compatibility, Kafka broker Like core, connect will use the log4j configuration file file (log4j2.properties) by default. it will show the following message also if it is in default modeBut for informational purpose, the following message will be shown when user launches kafka-server-start.sh:

DEPRECATED: using log4j 1.x configuration. To use log4j 2.x configuration, run with: 'export KAFKA_LOG4J_OPTS=\"-Dlog4j.configurationFile=file:$base_dir/../config/connect-log4j2.properties\"'

A log4j2 equivalent for traditional log4j config (connect-log4j2The test logging configuration (src/test/resources/log4j.properties) will be also provided, like other subprojects.

Add to this, the changes between log4j and log4j2 introduces the following changes:

• From log4j2, the name of the root logger becomes empty string from 'root'. It impacts Kafka connect's dynamic logging control feature. (And should be documented.)
• From log4j2, the logging level can't be null; GET `/admin/loggers` has been returning loggers with a non-null level (i.e., includes loggers with OFF level) but from log4j2, it returns the loggers with non-OFF levels only.
  
• If the user sets the root logger with PUT `/admin/loggers/{logger}`, all descendant loggers without the null level were affected. From log4j2, they will be affected since null level is regared as OFF level.

migrated into log4j2, also.

4. metadata

slf4j, log4j 1.x dependencies will be upgraded into log4j2 and the he test logging configuration (src/test/resources/log4j.properties) will be migrated into log4j2.

5. raft

Similar to connect and core. If the user launches test-kraft-server-start.sh, the following message will be shown:

DEPRECATED: using log4j 1.x configuration. To use log4j 2.x configuration, run with: 'export KAFKA_LOG4J_OPTS="-Dlog4j.configurationFile=file:$base_dir/../config/kraft-log4j2.properties"'

6. storage

slf4j, log4j 1.x dependencies will be upgraded into log4j2 and the he test logging configuration (src/test/resources/log4j.properties) will be migrated into log4j2.

7. streams

slf4j, log4j 1.x dependencies will be upgraded into log4j2 and the he Like core, the test logging configuration (src/test/resources/log4j.properties) will be updated migrated into log4j2.

...

.

...

Archetype log4j configuration will be updated into log4j2 equivalent (log4j2.properties).

5. Log4j-appender

Log4j-appender is the only module that does not affected by this update, since it provides a log4j 1.x appender.

Its log4j2 counterpart should be provided in near future. However, it is above the scope of this KIP.

6. Tools

Like core and connect, it will run with log4j configuration by default, but the users can use it by setting `export KAFKA_LOG4J_OPTS="-Dlog4j.configurationFile={log4j2-config-file-path}"`. A log4j2 equivalent for traditional log4j config (tools-log4j2.properties) will also be provided.

7. ETC

...

8. embedded zookeeper

Kafka provides an embedded zookeeper functionality with zookeeper-server-start.[sh|bat]. Since zookeeper's dynamic log level change feature depends on log4j 1.x (especially, Log4j MBean registration feature. see 'Run a JMX console' section here.), we will not support this functionality anymore. If the user runs zookeeper-server-start.[sh|bat], the following message will be displayed:

Running with log4j 2.x - Log4j MBean registration is not supported.

Compatibility, Deprecation, and Migration Plan

...

At some time or other,the default logging configuration format will be switched into log4j2. However, without removing log4j12-api it can also support log4j configuration format. In that point, The the informational message launcher scripts of core, connect, and raft will be also changed into like the following:

Code Block
language bash
Using log4j 2.x configuration. To use log4j 1.x configuration, run with: 'export KAFKA_LOG4J_OPTS="-Dlog4j.configuration=file:$base_dir/../config/log4j.properties"'

2. Client

None.

3. Connect

Same to core.

4. Strams

None.

5. Log4j-appender

This subproject will be remained for backward-compatibility, when the default logging format is changed into log4j2. Instead, its log4j2 counterpart, log4j2-appender, will be provided at that moment.

6. Tools

Same as core and connect.

7. ETC

...

Rejected Alternatives

None.