THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
Also, some tools always require config files for producer and consumer even when most people might just use default settings (ex. MirrorMaker).
FinallyRelated to this, some tools use a unique command line argument name for a command argument (ex. VerifyConsumerRebalance uses –zk to specify ZooKeeper URL while other tools use --zookeeper).
Public Interfaces
The following list summarizes the flavor of each tool we have now in Kafka. The argument style are marked in parentheses using the number in previous section.
ConsoleConsumer(1 & 2)
ConsoleProducer(2)
ConsumerOffsetChecker(2)
ConsumerPerformance(2)
DumpLogSegment(2)
ExportZkOffsets(2)
GetOffsetShell(2)
ImportZkOffsets(2)
JmxTool(2)
MirrorMaker(1)
ProducerPerformance - scala (2)
ProducerPerformance - java (3)
ReplayLogProducer(2)
ReplicaVerificationTool(2)
SimpleConsumerPerformance(2)
SimpleConsumerShell(2)
StateChangeLogMerger(2)
TestEndToEndLatency(3)
TestLogCleaning(2)
UpdateOffsetsInZk(3)
VerifyConsumerRebalacne(1)
Proposed Changes
Proposed Changes
- Change all tools using argument style #2 (arguments containing dots) to argument style #3 (arguments containing dashes) (see table below)Change ConsoleConsumer, VerifyConsumerRebalance tools to use style 2).
- (Suggested by Gwen) Since mirror maker is more like a service instead of a tool. It makes sense to pass in a mirror maker config file.
- Standardize Change unique/one-off command line argument naming by making the following argument name changes:names to match standard argument name (see table below).
| Command | Old Argument Name | New Argument Name |
|---|---|---|
| kafka-console-consumer.sh | consumer.config | consumer-config |
| kafka-consumer-offset-checker.sh | socket.timeout.ms | socket-timeout-ms |
| kafka-consumer-offset-checker.sh | retry.backoff.ms | retry-backoff-ms |
| ExportZkOffsets.scala | zkconnect | zookeeper |
| ImportZkOffsets.scala | zkconnect | zookeeper |
| KafkaMigrationTool.java | consumer.config | consumer-config |
| KafkaMigrationTool.java | producer.config | producer-config |
| KafkaMigrationTool.java | num.producers | num-producers |
| KafkaMigrationTool.java | num.streams | num-streams |
| KafkaMigrationTool.java | queue.size | queue-size |
| kafka-mirror-maker.sh | consumer.config | consumer-config |
| kafka-mirror-maker.sh | producer.config | producer-config |
| kafka-mirror-maker.sh | num.streams | num-streams |
| kafka-producer-perf-test.sh | request-num-acks | request-required-acks |
| kafka-replay-log-producer.sh | inputtopic | input-topic |
| kafka-replay-log-producer.sh | outputtopic | output-topic |
| kafka-replica-verification.sh | topic-white-list | whitelist |
| kafka-replica-verification.sh | report-interval-ms | reporting-interval |
| kafka-simple-consumer-shell.sh | fetchsize | fetch-size |
| TestLogCleaning.scala | zk | zookeeper |
| VerifyConsumerRebalance.scala | zookeeper.connect | zookeeper |
| TestLinearWriteSpeed.scala | compression | compression-codec |
| TestOffsetManager.scala | thread-count | threads |
| TestOffsetManager.scala | reporting-interval-ms | reporting-interval |
...
- What impact (if any) will there be on existing users? - Previously used command line arguments will no longer function and they will have to adapt their calls.
- If we are changing behavior how will we phase out the older behavior? - End-users will need to adaptThere will be no deprecation of old command line arguments, only a change-over.
- If we need special migration tools, describe them here. - None but note that No migration tools needed but the Kafka Documentation & Wiki pages will need to be updated to educate end-users on these changes.
- When will we remove the existing behavior? Users building from Existing behavior will change as soon as patch is committed and included in release build.
Rejected Alternatives
None