Current state: Accepted
Discussion thread: here
JIRA: here
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
The tools migration effort is ongoing and being tracked in KAFKA-14525. This is part of a bigger initiative to split the core module into multiple modules (e.g. storage, network, security, tools), which is being tracked in KAFKA-14524.
The plan is to migrate tools and related classes in a fully compatible way from kafka.tools and kafka.admin packages (core module) to org.apache.kafka.tools package (tools module).
While kicking off this activity, we identified a number of potential compatibility issues:
Detailed list of tools grouped by issue:
According to the KIP process, command line tools and arguments have to be considered public interface, so we want to maintain full compatibility or provide a deprecation period.
Rather than discussing common migration issues on a case by case basis, we would like to define some guidelines on how to address them in a consistent way.
Every tool should have its own wrapper script in bin and bin/windows directories. If a system test refers to the tool's FQCN, it should be updated to use the wrapper script name.
During the package deprecation period, we should provide a redirection to the new compatible implementation via reflection. We should also add a note in the release changes and create a sub-task as a reminder to remove the redirection in the next major release.
Example:
package kafka.tools @deprecated(since = "3.5") object JmxTool { def main(args: Array[String]): Unit = { println("WARNING: The 'kafka.tools' package is deprecated and will change to 'org.apache.kafka.tools' in the next major release.") val toolClass = Class.forName("org.apache.kafka.tools.JmxTool") val toolMethod = toolClass.getDeclaredMethod("main", classOf[Array[String]]) toolMethod.invoke(null, args) } } |
We only have one tool with this issue (kafka.tools.ConsoleProducer) and a number of known external implementations. We can address this as originally proposed in KIP-641.
We only have one tool with this issue (kafka.tools.StateChangeLogMerger), which is tracked in KAFKA-7735. This tool has been broken since 2.0 release, so it means that there is not much interest and we should simply deprecate.
However, the work for migrating and fixing this tool is on-going, but not included in the scope of this KIP.
Most tools communicate via Kafka protocol by using client classes, so they can be migrated without dependencies on core. Instead, tools with non-trivial dependencies on core (e.g. log message parsers) should wait until all required dependencies are migrated. If present, we should set related tasks as blockers for the tool migration sub-task.
Existing users will get a deprecation warning when using the old package name, while old SPIs and classes will be marked as deprecated.
All unit/integration tests should be migrated or added in case they are missing. All affected system tests have to pass with the new implementation.