Page History

Problem Statement

Eagle today does not have enough dynamic capabilities to support generic metric monitoring. Generic metric monitoring needs solve the following hard problems.

• Dynamically adding metrics in existing topology. It is common that one application may emit multiple metrics and the number of metrics may change. 
• Dynamically distributing metric and policy in existing topology. With multiple metrics and multiple policies per metric, even distribution of policy compute is critical
• Dynamically joining metrics. If one policy is for multiple metrics, we should make sure those multiple metrics come into this policy

Use Case Description

From Hadoop Native Metrics Monitoring, we can observed that 

• A typical metric stream should contain multiple metrics with different namenames. More often, these metrics are group grouped in one semantic category, say JMX metric, system metric.
• Multiple monitor logic monitoring rules are required based on different metric name/type.
• CEP logic becomes more and more complex when user need needs to do stream separation and join(need using pattern match).
• Un-named stream make the data flow logically route from spout to all next elements. Though the grouping helps here, with named stream, we could make route for the different metrics.

This page's content is recorded in 

Design

Data Flow & Generic Topology Structure

Multi-Plexer

The key idea is to split the metric stream into different streams, so that it could be routed to different (logic) executor. This introduce an a multiplexer. This multiplexer would be responsible to

• Understand the metric by metadata driven (a "metricName" field in the input tuple)
• Multiplex the metrics into different stream names (This is driven by the metric name/type)

Policy Executor

...

A route table is calculated and used by Multiplexer to decide how to route the given tuple. Basically, the route table ask answers the question "Given current tuple, which stream should the multiplexer route it to."

...

So that given m1 as current tuple, it would multiplex routed to stream1 and stream2 by look at the route table.

Metadata

The current policy definition would be extend extended to express the relationship between policy and metric. This would be used in MultiPlexer and Policy Executor to build the stream route map

{

...

   “policydef” : “…”

...

   “policydef” : “…”
   “Metrics”: [“m1”, “m2”]

...

 
},

...

{

   “policydef” : “…”

...

{
   “policydef” : “…”
   “Metrics”: [“m1”, “m3"]
}
]

}

Metric (Stream Schema):

{
    "name": "m1",
    "sourceTopic" : "hadoopJmxMetric",
    "fields": [
       {name: metricname, type:string},
       {name: timestamp, type:long},
       {name: value, type:double}
    ]
}

...

On boarding new metric flow

TBD