Page History

This is a FAQ for common questions that occur when debugging the operations of a running Flume cluster.

How can I get metrics from a node?

Flume nodes report metrics which you can use to debug and to see progress. You can look at a node's status web page by pointing your browser to port 35862. (http://<node>:35862).

How can I tell if data is arriving at the collector?

When events arrive at a collector, the source counters should be incremented on the node's metric page. For example, if you have a node called foo you should see the following fields have growing values when you refresh the page.

• LogicalNodeManager.foo.source.CollectorSource.number of bytes
• LogicalNodeManager.foo.source.CollectorSource.number of events

How can I tell if data is being written to HDFS?

Data in hdfs doesn't "arrive" in hdfs until the file is closed or certain size thresholds are met. To see progress, you can look at the status web page on the collector node on port 35862. (http://<collector>:35862). If data is arriving if the the ...As events are written to hdfs, the sink counters on the collector's metric page should be incrementing. In particular look for fields that match the following names:

• *.Collector.GunzipDecorator.UnbatchingDecorator.AckChecksumChecker.InsistentAppend.append*

*.appendSuccesses are successful writes. If other values like appendRetries or appendGiveups are incremented, they indicate a problem with the attemps to write.appendSuccessses metric should be incrementing.

I am getting a lot of duplicated event data. Why is this happening and what can I do to make this go away?

...