THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
Status
Current state: Under Discussion
Discussion thread: TBD
JIRA:
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation
Currently tools in Kafka have three flavor of command line arguments:
- command line arguments using dot, e.g. --consumer.config
- command line arguments using dash --broker-list
- command line arguments using position
#2 and #3 conforms to Unix standards while #1 does not. We want to standardize them.
Also, some tools always require config files for producer and consumer even when most people might just use default settings (ex. MirrorMaker).
Related to this, some tools use a unique command line argument name for a command argument (ex. VerifyConsumerRebalance uses –zk to specify ZooKeeper URL while other tools use --zookeeper).
Proposed Changes
- Change all tools using argument style #2 (arguments containing dots) to argument style #3 (arguments containing dashes) (see table below).
- (Suggested by Gwen) Since mirror maker is more like a service instead of a tool. It makes sense to pass in a mirror maker config file.
- Change unique/one-off command line argument names to match standard argument name (see table below).
| Command | Old Argument Name | New Argument Name |
|---|---|---|
| kafka-console-consumer.sh | consumer.config | consumer-config |
| kafka-consumer-offset-checker.sh | socket.timeout.ms | socket-timeout-ms |
| kafka-consumer-offset-checker.sh | retry.backoff.ms | retry-backoff-ms |
| ExportZkOffsets.scala | zkconnect | zookeeper |
| ImportZkOffsets.scala | zkconnect | zookeeper |
| KafkaMigrationTool.java | consumer.config | consumer-config |
| KafkaMigrationTool.java | producer.config | producer-config |
| KafkaMigrationTool.java | num.producers | num-producers |
| KafkaMigrationTool.java | num.streams | num-streams |
| KafkaMigrationTool.java | queue.size | queue-size |
| kafka-mirror-maker.sh | consumer.config | consumer-config |
| kafka-mirror-maker.sh | producer.config | producer-config |
| kafka-mirror-maker.sh | num.streams | num-streams |
| kafka-producer-perf-test.sh | request-num-acks | request-required-acks |
| kafka-replay-log-producer.sh | inputtopic | input-topic |
| kafka-replay-log-producer.sh | outputtopic | output-topic |
| kafka-replica-verification.sh | topic-white-list | whitelist |
| kafka-replica-verification.sh | report-interval-ms | reporting-interval |
| kafka-simple-consumer-shell.sh | fetchsize | fetch-size |
| TestLogCleaning.scala | zk | zookeeper |
| VerifyConsumerRebalance.scala | zookeeper.connect | zookeeper |
| TestLinearWriteSpeed.scala | compression | compression-codec |
| TestOffsetManager.scala | thread-count | threads |
| TestOffsetManager.scala | reporting-interval-ms | reporting-interval |
Note that the following additional command line argument standardizations will be tracked via KAFA-1293 but are not part of this KIP since they are less invasive:
- Add 'help' command line argument to any top-level tool scripts that were missing it.
- Properly format argument descriptions (into sentences) and add any missing "REQUIRED" notes.
Compatibility, Deprecation, and Migration Plan
- What impact (if any) will there be on existing users? - Previously used command line arguments will no longer function and they will have to adapt their calls.
- If we are changing behavior how will we phase out the older behavior? - There will be no deprecation of old command line arguments, only a change-over.
- If we need special migration tools, describe them here. - No migration tools needed but the Kafka Documentation & Wiki pages will need to be updated to educate end-users on these changes.
- When will we remove the existing behavior? Existing behavior will change as soon as patch is committed and included in release build.
Rejected Alternatives
None