THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
e.g kafka-console-producer.sh --brokerbootstrap-list localhostserver `hostname`:9092 --topic my_topic < my_filetest-01 < 1.txt (Only supports reading a single file)
Or use a more complex shell to enter the contents of multiple files into kafka.
As far as user habits are concerned, when encountering such problems, they always ask Kafka whether it supports such use at the first time, and then look for other alternative solutions.
If the application layer of the kafka producer command line tool supports reading multi-file data, it will be more convenient for users.
Even if we still only need to read the data of a single file, at least from the application layer to support this function, there will be more choices for users, which does not do any harm.
On the other hand, I want to separate the multi-file data reading from the shell, so that the kafka tool itself has this function.
On different operating system platforms, using different interpreters, reading multi-file scripts needs to consider compatibility, such as the difference between bash and bat.
If the producer command line tool naturally supports reading multiple files, it will be very convenient for users to use this function on the windows platform without learning bat.
So We want to introduce --files to the producer command line tool to support reading data from a given multi-file,
...
kafka-console-producer.sh:
The producer command will accept --files and --files-separator optionsoption.
E.g
1.The content in the 1.txt file is:
a
b
c
...
The current implementation scheme is not complicated, it does not have much difference with Line MessageReader, but in order to distinguish it from Line, I extracted FileMessageReader.
MessageReader
...
| language | scala |
|---|---|
| title | MessageReader |
...
FileMessageReader
...
| language | scala |
|---|---|
| title | FileMessageReader |
...
FileMessageReader will be used to detect the availability of the input file list and read multiple files.
Compatibility, Deprecation, and Migration Plan
The new options have no effect on existing usage.
Rejected Alternatives
None1.Discard the --files-separator option: The comma separator conforms to the habit of most users, and this habit does not change frequently. Similarly, we will use the comma as the default separator and just follow this rule.