...

JIRA:

Motivation

More discussions: https://github.com/apache/kafka/pull/16260#issuecomment-2159632052

The logging library used in the tools module is inconsistent with the core. Now, slf4j2.x provides a cleaner and more straightforward approach.

By adding slf4j backends to dependencies, distributions will have their slf4j backends, and it is safe to define a slf4j provider in kafka-run-class. This also allows running a Kafka instance from the source code with a specific provider.

More importantly, this change will bring the following benefits:

• Specifying the provider class explicitly via the slf4j.provider  system property bypasses the service loader mechanism for finding providers and may shorten SLF4J initialization. (reference: https://www.slf4j.org/faq.html#explicitProvider)
• Indirectly controlling the logging frameworks that Kafka supports by explicitly declaring supported SLF4J providers. This approach would require users to identify and implement logging frameworks that are compatible with our declared providers. By implementing this strategy, we can substantially reduce our maintenance overhead while ensuring consistent logging behavior across implementations.

• Our current build configuration employs fragile dependency management tricks to handle SLF4J backends. We can eliminate these brittle build mechanisms by transitioning to explicit provider dependencies after upgrading to 2.0. This would create a more robust system where users can override logging configurations as needed, greatly simplifying our maintenance process and the user experience.

Kafka still relies on SLF4J1 as its logging interface, significantly restricting users when selecting logging backends. Users must either ensure their preferred framework appears first in the classpath order or remove other logging frameworks entirely—both approaches require manual JAR file manipulation.

With the introduction of SLF4J2, a new system property -Dslf4j.provider becomes available, allowing users to select logging backends through simple configuration rather than modifying JAR files. 

Note: The rationale for this KIP is that upgrading SLF4J necessitates corresponding provider upgrades, which constitutes a breaking change and should be delivered in Kafka 5.0

More discussions: https://github.com/apache/kafka/pull/16260#issuecomment-2159632052

Public Interfaces

Update the below files to upgrade slf4j

• dependencies.gradle 
• build.gradle 
• kafka-run-class 

Proposed Changes

1. Upgrade slf4j from 1.7.36 to 2.0.17
   • We will also upgrade the log4j2 dependency based on SLF4J2 (i.e.,  log4j-slf4j-impl to log4j-slf4j2-impl ) 
2. Add a new system variable to the kafka-run-class  (sh & bat) script to define -Dslf4j.provider.
   • By default, we use org.apache.logging.slf4j.SLF4JServiceProvider 
   • reference: https://www.slf4j.org/faq.html#explicitProvider
3. Add other popular slf4j backend binding provider dependencies.
   • log4j2 (project default) 
   • logback 
   • java.util.logging 
4. Define the default logging framework for Kafka
   • Log4j2 will be the standard server-side logging framework
   • We expose the log4j2 configuration file (log4j2.properties) in the Kafka distribution for users to customize

Compatibility, Deprecation, and Migration Plan

Note: As we introduce this change in version 4.1.0, which is a minor version, we must ensure it doesn't break existing users' code. If backward compatibility cannot be guaranteed, this change will be postponed to version 5.0.0. 

Ensure the user's code works well when specifying their preferred logging frameworks.

We will provide compatible binding jars for log4j2 in Kafka when upgrading slf4j-api  to 2.0.17.

Since we provide the backend binding dependencies, users should be able to upgrade their backend dependencies seamlessly.

Please note that users have to upgrade their logging framework to the matched provider explicitly, and we expect users to be responsible for maintaining consistent versions when using other providers.

...

• We will also upgrade the log4j2 dependency based on SLF4J2 
• To avoid breaking downstream compatibility, we will not bind the server side to any specific logging framework in the build.gradle, maintaining the current scope - compileOnly  and releaseOnly 
  • https://github.com/apache/kafka/pull/17373#issuecomment-2577813317
• For client-side components, users will need to select their own logging framework, as client-side code typically runs with user applications and should use the same logging framework
• For server-side components, users who wish to use a different logging framework will need to add appropriate jars to the classpath and configure the -Dslf4j.provider  property accordingly
• Version matching between the SLF4J core and providers is critical, as documented in the SLF4J FAQ (https://www.slf4j.org/faq.html#compatibility). Users must ensure the provider version is compatible with SLF4J 2.0.17 when they select their logging framework. 

Test Plan

Imitate the user's behavior by manually adding or replacing the dependent logging jars and assigning the corresponding system property by -Dslf4j.provider .
After that, try starting a Kafka service and see if there are any warnings or errors. Make sure Kafka still usually works after changing the logging framework.

Rejected Alternatives

...

• Delivering this KIP in a minor version
  
  • We cannot ensure SLF4J 2.x would work like 1.x out of the box for client modules. This would potentially break backward compatibility, which violates Kafka's commitment to compatibility in minor releases.
  • Solutions involving warnings and classpath ordering do not provide robust solutions to compatibility challenges. Such approaches would introduce fragility and unpredictability to deployment environments, especially in complex production settings where Kafka is commonly used.
  • Version matching between the SLF4J core and providers is critical, as documented in the SLF4J FAQ (https://www.slf4j.org/faq.html#compatibility). A major version change like this requires careful coordination that is more appropriate for a major release.