Versions Compared

Old Version 2

changes.mady.by.user Jonathan Hsieh

Saved on

compared with

New Version 3

changes.mady.by.user Jonathan Hsieh

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Let's see the see what this metrics json looks like for an agent node and for a collector node.

Collectors

Here's an example of a node setup to be a collector. This is very verbose and provides all metrics for all of the stages of the flume pipeline.

Code Block

{ "hostname" : "grimlock",
  "name" : "grimlock",
  "nodename" : "grimlock",
  "physicalnode" : "grimlock",
  "reconfigures" : 2,
  "sink.Collector.GunzipDecorator.UnbatchingDecorator.AckChecksumChecker.InsistentAppend.StubbornAppend.InsistentOpen.MaskDecorator.Roll.RollDetectDeco.EscapedCustomDfsSink.name" : "EscapedCustomDfsSink",
  "sink.Collector.GunzipDecorator.UnbatchingDecorator.AckChecksumChecker.InsistentAppend.StubbornAppend.InsistentOpen.MaskDecorator.Roll.RollDetectDeco.name" : "RollDetectDeco",
  "sink.Collector.GunzipDecorator.UnbatchingDecorator.AckChecksumChecker.InsistentAppend.StubbornAppend.InsistentOpen.MaskDecorator.Roll.name" : "Roll",
  "sink.Collector.GunzipDecorator.UnbatchingDecorator.AckChecksumChecker.InsistentAppend.StubbornAppend.InsistentOpen.MaskDecorator.Roll.rollfails" : 0,
  "sink.Collector.GunzipDecorator.UnbatchingDecorator.AckChecksumChecker.InsistentAppend.StubbornAppend.InsistentOpen.MaskDecorator.Roll.rolls" : 0,
  "sink.Collector.GunzipDecorator.UnbatchingDecorator.AckChecksumChecker.InsistentAppend.StubbornAppend.InsistentOpen.MaskDecorator.Roll.rollspec" : "escapedCustomDfs(\"file:///tmp/test\",\"foo%{rolltag}\" )",
  "sink.Collector.GunzipDecorator.UnbatchingDecorator.AckChecksumChecker.InsistentAppend.StubbornAppend.InsistentOpen.MaskDecorator.name" : "MaskDecorator",
  "sink.Collector.GunzipDecorator.UnbatchingDecorator.AckChecksumChecker.InsistentAppend.StubbornAppend.InsistentOpen.backoffPolicy.CumulativeCappedExpBackoff.backoffCount" : 0,
  "sink.Collector.GunzipDecorator.UnbatchingDecorator.AckChecksumChecker.InsistentAppend.StubbornAppend.InsistentOpen.backoffPolicy.CumulativeCappedExpBackoff.backoffCurrentMs" : 1000,
  "sink.Collector.GunzipDecorator.UnbatchingDecorator.AckChecksumChecker.InsistentAppend.StubbornAppend.InsistentOpen.backoffPolicy.CumulativeCappedExpBackoff.backoffInitialMs" : 1000,
  "sink.Collector.GunzipDecorator.UnbatchingDecorator.AckChecksumChecker.InsistentAppend.StubbornAppend.InsistentOpen.backoffPolicy.CumulativeCappedExpBackoff.backoffMaxCumulativeMs" : 2147483647,
  "sink.Collector.GunzipDecorator.UnbatchingDecorator.AckChecksumChecker.InsistentAppend.StubbornAppend.InsistentOpen.backoffPolicy.CumulativeCappedExpBackoff.backoffRetryTime" : 1311640637955,
  "sink.Collector.GunzipDecorator.UnbatchingDecorator.AckChecksumChecker.InsistentAppend.StubbornAppend.InsistentOpen.backoffPolicy.CumulativeCappedExpBackoff.backoffSleepCapMs" : 60000,
  "sink.Collector.GunzipDecorator.UnbatchingDecorator.AckChecksumChecker.InsistentAppend.StubbornAppend.InsistentOpen.backoffPolicy.CumulativeCappedExpBackoff.name" : "CumulativeCappedExpBackoff",
  "sink.Collector.GunzipDecorator.UnbatchingDecorator.AckChecksumChecker.InsistentAppend.StubbornAppend.InsistentOpen.name" : "InsistentOpen",
  "sink.Collector.GunzipDecorator.UnbatchingDecorator.AckChecksumChecker.InsistentAppend.StubbornAppend.InsistentOpen.openAttempts" : 1,
  "sink.Collector.GunzipDecorator.UnbatchingDecorator.AckChecksumChecker.InsistentAppend.StubbornAppend.InsistentOpen.openGiveups" : 0,
  "sink.Collector.GunzipDecorator.UnbatchingDecorator.AckChecksumChecker.InsistentAppend.StubbornAppend.InsistentOpen.openRequests" : 1,
  "sink.Collector.GunzipDecorator.UnbatchingDecorator.AckChecksumChecker.InsistentAppend.StubbornAppend.InsistentOpen.openRetries" : 0,
  "sink.Collector.GunzipDecorator.UnbatchingDecorator.AckChecksumChecker.InsistentAppend.StubbornAppend.InsistentOpen.openSuccessses" : 1,
  "sink.Collector.GunzipDecorator.UnbatchingDecorator.AckChecksumChecker.InsistentAppend.StubbornAppend.appendFails" : 0,
  "sink.Collector.GunzipDecorator.UnbatchingDecorator.AckChecksumChecker.InsistentAppend.StubbornAppend.appendRecovers" : 0,
  "sink.Collector.GunzipDecorator.UnbatchingDecorator.AckChecksumChecker.InsistentAppend.StubbornAppend.appendSuccess" : 0,
  "sink.Collector.GunzipDecorator.UnbatchingDecorator.AckChecksumChecker.InsistentAppend.StubbornAppend.name" : "StubbornAppend",
  "sink.Collector.GunzipDecorator.UnbatchingDecorator.AckChecksumChecker.InsistentAppend.appendAttempts" : 0,
  "sink.Collector.GunzipDecorator.UnbatchingDecorator.AckChecksumChecker.InsistentAppend.appendGiveups" : 0,
  "sink.Collector.GunzipDecorator.UnbatchingDecorator.AckChecksumChecker.InsistentAppend.appendRequests" : 0,
  "sink.Collector.GunzipDecorator.UnbatchingDecorator.AckChecksumChecker.InsistentAppend.appendRetries" : 0,
  "sink.Collector.GunzipDecorator.UnbatchingDecorator.AckChecksumChecker.InsistentAppend.appendSuccessses" : 0,
  "sink.Collector.GunzipDecorator.UnbatchingDecorator.AckChecksumChecker.InsistentAppend.backoffPolicy.CumulativeCappedExpBackoff.backoffCount" : 0,
  "sink.Collector.GunzipDecorator.UnbatchingDecorator.AckChecksumChecker.InsistentAppend.backoffPolicy.CumulativeCappedExpBackoff.backoffCurrentMs" : 1000,
  "sink.Collector.GunzipDecorator.UnbatchingDecorator.AckChecksumChecker.InsistentAppend.backoffPolicy.CumulativeCappedExpBackoff.backoffInitialMs" : 1000,
  "sink.Collector.GunzipDecorator.UnbatchingDecorator.AckChecksumChecker.InsistentAppend.backoffPolicy.CumulativeCappedExpBackoff.backoffMaxCumulativeMs" : 2147483647,
  "sink.Collector.GunzipDecorator.UnbatchingDecorator.AckChecksumChecker.InsistentAppend.backoffPolicy.CumulativeCappedExpBackoff.backoffRetryTime" : 1311640637904,
  "sink.Collector.GunzipDecorator.UnbatchingDecorator.AckChecksumChecker.InsistentAppend.backoffPolicy.CumulativeCappedExpBackoff.backoffSleepCapMs" : 60000,
  "sink.Collector.GunzipDecorator.UnbatchingDecorator.AckChecksumChecker.InsistentAppend.backoffPolicy.CumulativeCappedExpBackoff.name" : "CumulativeCappedExpBackoff",
  "sink.Collector.GunzipDecorator.UnbatchingDecorator.AckChecksumChecker.InsistentAppend.name" : "InsistentAppend",
  "sink.Collector.GunzipDecorator.UnbatchingDecorator.AckChecksumChecker.ackEnds" : 0,
  "sink.Collector.GunzipDecorator.UnbatchingDecorator.AckChecksumChecker.ackFails" : 0,
  "sink.Collector.GunzipDecorator.UnbatchingDecorator.AckChecksumChecker.ackStarts" : 0,
  "sink.Collector.GunzipDecorator.UnbatchingDecorator.AckChecksumChecker.ackSuccesses" : 0,
  "sink.Collector.GunzipDecorator.UnbatchingDecorator.AckChecksumChecker.ackUnexpected" : 0,
  "sink.Collector.GunzipDecorator.UnbatchingDecorator.AckChecksumChecker.name" : "AckChecksumChecker",
  "sink.Collector.GunzipDecorator.UnbatchingDecorator.batchCount" : 0,
  "sink.Collector.GunzipDecorator.UnbatchingDecorator.name" : "UnbatchingDecorator",
  "sink.Collector.GunzipDecorator.UnbatchingDecorator.passthroughCount" : 0,
  "sink.Collector.GunzipDecorator.UnbatchingDecorator.unbatchedCount" : 0,
  "sink.Collector.GunzipDecorator.gunzippedSize" : 0,
  "sink.Collector.GunzipDecorator.gzippedCount" : 0,
  "sink.Collector.GunzipDecorator.gzippedSize" : 0,
  "sink.Collector.GunzipDecorator.name" : "GunzipDecorator",
  "sink.Collector.GunzipDecorator.passthroughCount" : 0,
  "sink.Collector.name" : "Collector",
  "sinkConfig" : "collectorSink( \"file:///tmp/test\", \"foo\" )",
  "source.CollectorSource.name" : "CollectorSource",
  "source.CollectorSource.number of bytes" : 0,
  "source.CollectorSource.number of events" : 0,
  "source.CollectorSource.type" : "CollectorSource",
  "sourceConfig" : "collectorSource",
  "state" : "ACTIVE",
  "version" : "Mon Jul 25 17:37:13 PDT 2011"
}

Each stage of a flume sink pipeline is laid out in "dot" notation.

The most interesting fields to follow are likely:

  • "source.CollectorSource.number of bytes" - number of bytes accepted by source
  • "source.CollectorSource.number of events" - number of events accepted by source
  • "sink.Collector.GunzipDecorator.UnbatchingDecorator.AckChecksumChecker.InsistentAppend.appendAttempts" - number of events sink attempted to write to subsink including retries (TODO: check)
  • "sink.Collector.GunzipDecorator.UnbatchingDecorator.AckChecksumChecker.InsistentAppend.appendGiveups" - number of events that cause an irrecoverable error when attempting to write
  • "sink.Collector.GunzipDecorator.UnbatchingDecorator.AckChecksumChecker.InsistentAppend.appendRequests" - number of
    events sink was acked to write (TODO: check)
  • "sink.Collector.GunzipDecorator.UnbatchingDecorator.AckChecksumChecker.InsistentAppend.appendRetries" - number of of times any event has been retried.
  • "sink.Collector.GunzipDecorator.UnbatchingDecorator.AckChecksumChecker.InsistentAppend.appendSuccessses" - number of event successfully written by sink.

Agents

BE mode

DFO mode

E2E mode

Failover Chains

Masters