...
Also, some tools always require config files for producer and consumer even when most people might just use default settings (ex. MirrorMaker).
FinallyRelated to this, some tools use a unique command line argument name for a command argument (ex. VerifyConsumerRebalance uses –zk to specify ZooKeeper URL while other tools use --zookeeper).
Public Interfaces
The following list summarizes the flavor of each tool we have now in Kafka. The argument style are marked in parentheses using the number in previous section.
ConsoleConsumer(1 & 2)
ConsoleProducer(2)
ConsumerOffsetChecker(2)
ConsumerPerformance(2)
DumpLogSegment(2)
ExportZkOffsets(2)
GetOffsetShell(2)
ImportZkOffsets(2)
JmxTool(2)
MirrorMaker(1)
ProducerPerformance - scala (2)
ProducerPerformance - java (3)
ReplayLogProducer(2)
ReplicaVerificationTool(2)
SimpleConsumerPerformance(2)
SimpleConsumerShell(2)
StateChangeLogMerger(2)
TestEndToEndLatency(3)
TestLogCleaning(2)
UpdateOffsetsInZk(3)
VerifyConsumerRebalacne(1)
Proposed Changes
Proposed Changes
- Change all tools using argument style #2 (arguments containing dots) to argument style #3 (arguments containing dashes) (see table below)Change ConsoleConsumer, VerifyConsumerRebalance tools to use style 2).
- (Suggested by Gwen) Since mirror maker is more like a service instead of a tool. It makes sense to pass in a mirror maker config file.
- Standardize Change unique/one-off command line argument naming by making the following argument name changes:names to match standard argument name (see table below).
Command | Old Argument Name | New Argument Name |
---|---|---|
kafka-console-consumer.sh | consumer.config | consumer-config |
kafka-consumer-offset-checker.sh | socket.timeout.ms | socket-timeout-ms |
kafka-consumer-offset-checker.sh | retry.backoff.ms | retry-backoff-ms |
ExportZkOffsets.scala | zkconnect | zookeeper |
ImportZkOffsets.scala | zkconnect | zookeeper |
KafkaMigrationTool.java | consumer.config | consumer-config |
KafkaMigrationTool.java | producer.config | producer-config |
KafkaMigrationTool.java | num.producers | num-producers |
KafkaMigrationTool.java | num.streams | num-streams |
KafkaMigrationTool.java | queue.size | queue-size |
kafka-mirror-maker.sh | consumer.config | consumer-config |
kafka-mirror-maker.sh | producer.config | producer-config |
kafka-mirror-maker.sh | num.streams | num-streams |
kafka-producer-perf-test.sh | request-num-acks | request-required-acks |
kafka-replay-log-producer.sh | inputtopic | input-topic |
kafka-replay-log-producer.sh | outputtopic | output-topic |
kafka-replica-verification.sh | topic-white-list | whitelist |
kafka-replica-verification.sh | report-interval-ms | reporting-interval |
kafka-simple-consumer-shell.sh | fetchsize | fetch-size |
TestLogCleaning.scala | zk | zookeeper |
VerifyConsumerRebalance.scala | zookeeper.connect | zookeeper |
TestLinearWriteSpeed.scala | compression | compression-codec |
TestOffsetManager.scala | thread-count | threads |
TestOffsetManager.scala | reporting-interval-ms | reporting-interval |
...
- What impact (if any) will there be on existing users? - Previously used command line arguments will no longer function and they will have to adapt their calls.
- If we are changing behavior how will we phase out the older behavior? - End-users will need to adaptThere will be no deprecation of old command line arguments, only a change-over.
- If we need special migration tools, describe them here. - None but note that No migration tools needed but the Kafka Documentation & Wiki pages will need to be updated to educate end-users on these changes.
- When will we remove the existing behavior? Users building from Existing behavior will change as soon as patch is committed and included in release build.
Rejected Alternatives
None