Table of Contents |
---|
Status
Current state: Under Discussion
Discussion thread: here
Status | ||||
---|---|---|---|---|
|
Discussion thread: here
Vote thread: here
JIRA:
Jira | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|
|
Pull Request: hereJIRA:
Motivation
Tools kafka-console-producer.sh
and kafka-console-consumer.sh
have the multiple option --property
to setup the MessageReader
for Producer (defined by --line-reader
) or the MessageFormatter
for Consumer (defined by --formatter
).
...
The idea is to offer the same possibility to use a config file option to setup the MessageReader
/MessageFormatter
in addition to the existing option --property
.
It will be useful in different use cases:
...
Public Interfaces
Add an option:
--reader-config
to t
...
- ool
kafka-console-producer.sh
...
--formatter-config
to toolkafka-console-consumer.sh
Alternative: As for --reader.config
for Producer and --formatter.config
for Consumer, so it is less generic (but on the other hand, the current --property
is quite generic already) and cannot be confused with other config producer-property
/--consumer-property
with --consumer.config
/--producer.config
, any value from option --property
would override value from config file.
As for --producer-property
/--consumer-property
with --consumer.config
/--producer.config
, any value from option --property
would override value from config file.
Proposed Changes
Simple example
Before (multiple
--property
options):Code Block language bash kafka-console-producer.sh [...] --property print.timestamp=true \ --property print.key=true \ --property print.offset=true \ --property print.partition=true \ --property print.headers=true \ --property print.value=true \ --property key.separator=':' \ --property line.separator=';' \ --property headers.separator='|'
After (all properties in one file):
Code Block language actionscript3 title reader-config.properties print.timestamp=true print.key=true print.offset=true print.partition=true print.headers=true print.value=true key.separator=: line.separator=; headers.separator=|
Code Block language bash kafka-console-producer.sh [...] --reader-config reader-config.properties
Mixed example
When a property is in the config file and as option, option wins.
Code Block | ||||
---|---|---|---|---|
| ||||
print.timestamp=true
print.key=true
print.offset=true
print.partition=true
print.headers=true
print.value=true
key.separator=:
line.separator=;
headers.separator=| |
Code Block | ||
---|---|---|
| ||
kafka-console-producer.sh [...] --reader-config reader-config.properties --property key.separator='/' |
Then, the MessageReader
will get key.separator=/
, not key.separator=:
Proposed Changes
Implement the new options as described in Public Interfaces section:
ConsoleProducer
: new option "reader-config
" with description "Config properties file for the message reader. Note that --property takes precedence over this config.
"ConsoleConsumer
: new option "formatter-config
" with description "Config properties file to initialize the message formatter. Note that --property takes precedence over this config.
"
Already implemented in pull request #12175.
...
Define a property in config file and in options and check that option override config file.
Rejected Alternatives
...
- Use of generic option
--config
to avoid confusion in the purpose of this config file (but on the other hand, the current--property
is quite generic already, maybe another KIP to change it). - Use of option name
--read.config
/--formatter.config
, with ".
" as separator, as it's not consitent with the commonly used option--command-config