Status
Current state: Accepted
Discussion thread: here
JIRA: KAFKA-6733
Motivation
ConsoleConsumer (kafka-console-consumer) is a very important debugging tool in Kafka. However, currently it cannot print offset, partition and headers of a Kafka record. So, during debugging session, if we need to know those information, we have to use kafka-dump-log, which require file system access to the Kafka broker hosts, or use custom application, like kafkacat. This KIP is proposing to rectify this issue.
Currently, without any extra property, ConsoleConsumer prints the value of the record.
And the following properties control the ConsoleConsumer output format:
As you see, there is no way to print message offset, partition and headers. Also, there is no easy way to differentiate null value or blank string.
Public Interfaces
New properties to DefaultMessageFormatter
which can be specified in kafka-console-consumer
command line arguments with --property
argument.
Property | Valid Values | Default Value | Description |
---|---|---|---|
print.offset | "true" or "false" | "false" | print message offset |
print.headers | "true" or "false" | "false" | print headers, will print NO_HEADERS if the record has no headers |
header.separator | string | "," | separator printed between each header's key:value pair |
headers.deserializer | class name | "StringDeserializer" | header value deserializer |
null.literal | string | "null" | literal to print if the value is NULL (size -1) |
Example output:
$ kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning --property print.partition=true --property print.key=true --property print.timestamp=true --property print.offset=true --property print.headers=true --property key.separator='|' CreateTime:1592475472398|Partition:0|Offset:3|h1:v1,h2:v2|key1|value1 CreateTime:1592475472456|Partition:0|Offset:4|NO_HEADERS|key2|value2 |
Add new properties to DefaultMessageFormatter
as per the above:
This KIP introduces incompatibility when "print.partition=true" (existing argument). Before this KIP, it prints the partition as a number after the value. After this KIP, it prints the partition prefixed with "Partition:", like "Partition:0" before the key (if printed) and value.
The other changes are backward compatible because they do not exist before. Apart from "print.partition=true", if a user does not use any new parameters, then the output of console consumer will look the same as before.