Motivation
As a system Kafka has a fairly simple description of correctness: all messages sent to a topic should be delivered to each consumer group consuming that topic. The goal of this facility is to continuously monitor that this is occurring. A facility similar to this exists at LinkedIn.
Audit Messages
Clients that support auditing do so by periodically reporting a summary of the messages they have sent or received. These messages are then aggregated to check that all messages sent were received. Auditing is an optional facility--clients that don't support it simple will not be able to use the audit monitoring tool.
...
Code Block |
---|
{
"guid": "51656274-a86a-4dff-b824-8e8e20a6348f", // a unique identifier for this audit message
"time": 1348335709389, // time at which the audit began
"host": "host-name", // the host name of the machine sending the audit
"facility": "datacenter-name", // the data center of the machine sending the audit
"client": "application-name", // the client application name
"tier":"producer", // the name of the logical tier (producer, broker, or other consumer group)
"offset_begin": 22157000, // the beginning offset this covers
"offset_end": 22239000, // the ending offset this covers
"summary": 1234234 // a hash summary of the messages seen.
}
|
General Cluster UI
There are several uses for a web UI related to Kafka, we may as well fold these into a single tool. We did a code dump of the current audit tool as part of KAFKA-260, but if we are going to expand the functionality it might make sense to clean up that code, and move it into Scala.
...