Status

Current state: Under Discussion

Discussion thread: here

Vote thread: here

JIRA:

Motivation

More discussions: https://github.com/apache/kafka/pull/16260#issuecomment-2159632052

The logging library used in the tools module is inconsistent with the core. Now, slf4j2.x provides a cleaner and more straightforward approach.

By adding slf4j backends to dependencies, distributions will have their slf4j backends, and it is safe to define a slf4j provider in kafka-run-class. This also allows running a Kafka instance from the source code with a specific provider.

More importantly, this change will bring the following benefits:

• Specifying the provider class explicitly via the slf4j.provider  system property bypasses the service loader mechanism for finding providers and may shorten SLF4J initialization. (reference: https://www.slf4j.org/faq.html#explicitProvider)
• Indirectly controlling the logging frameworks that Kafka supports by explicitly declaring supported SLF4J providers. This approach would require users to identify and implement logging frameworks that are compatible with our declared providers. By implementing this strategy, we can substantially reduce our maintenance overhead while ensuring consistent logging behavior across implementations.

• Our current build configuration employs fragile dependency management tricks to handle SLF4J backends. We can eliminate these brittle build mechanisms by transitioning to explicit provider dependencies after upgrading to 2.0. This would create a more robust system where users can override logging configurations as needed, greatly simplifying our maintenance process and the user experience.

Public Interfaces

Update the below files to upgrade slf4j

• dependencies.gradle 

• build.gradle 

Proposed Changes

1. Upgrade slf4j from 1.7.36 to 2.0.17
2. Add a new system variable to the kafka-run-class  (sh & bat) script to define -Dslf4j.provider.
   • By default, we use org.apache.logging.slf4j.SLF4JServiceProvider 
   • reference: https://www.slf4j.org/faq.html#explicitProvider
3. Add other popular slf4j backend binding provider dependencies, e.g., logback 

Compatibility, Deprecation, and Migration Plan

Note: As we introduce this change in version 4.1.0, which is a minor version, we must ensure it doesn't break existing users' code. If backward compatibility cannot be guaranteed, this change will be postponed to version 5.0.0. 

Ensure the user's code works well when specifying their preferred logging frameworks.

We will provide compatible binding jars for log4j2 in Kafka when upgrading slf4j-api  to 2.0.17.

Since we provide the backend binding dependencies, users should be able to upgrade their backend dependencies seamlessly.

Please note that users have to upgrade their logging framework to the matched provider explicitly, and we expect users to be responsible for maintaining consistent versions when using other providers.

For future slf4j upgrade: In the future, for slf4j upgrades, we agree not to have a KIP again and follow the same approach mentioned in this KIP.

Test Plan

Imitate the user's behavior by manually adding or replacing the dependent logging jars and assigning the corresponding system property by -Dslf4j.provider .
After that, try starting a Kafka service and see if there are any warnings or errors. Make sure Kafka still usually works after changing the logging framework.

Rejected Alternatives

N/A