Problem Statement
Eagle today does not have enough dynamic capabilities to support generic metric monitoring. Generic metric monitoring needs solve the following hard problems.
- Dynamically adding metrics in existing topology. It is common that one application may emit multiple metrics and the number of metrics may change.
- Dynamically distributing metric and policy in existing topology. With multiple metrics and multiple policies per metric, even distribution of policy compute is critical
- Dynamically joining metrics. If one policy is for multiple metrics, we should make sure those multiple metrics come into this policy
Use Case Description
From Hadoop Native Metrics Monitoring, we can observed that
- A typical metric stream should contain multiple metrics with different names. More often, these metrics are grouped in one semantic category, say JMX metric, system metric.
- Multiple monitoring rules are required based on different metric name/type.
- CEP logic becomes more and more complex when user needs to do stream separation and join(need using pattern match).
- Un-named stream make the data flow logically route from spout to all next elements. Though the grouping helps here, with named stream, we could make route for the different metrics.
This page's content is recorded in
Jira | ||||||
---|---|---|---|---|---|---|
|
Design
Data Flow & Generic Topology Structure
Gliffy Diagram | ||||
---|---|---|---|---|
|
Multi-Plexer
The key idea is to split the metric stream into different streams, so that it could be routed to different (logic) executor. This introduce a multiplexer. This multiplexer would be responsible to
...
So that given m1 as current tuple, it would multiplex routed to stream1 and stream2 by look at the route table.
Metadata
The current policy definition would be extended to express the relationship between policy and metric. This would be used in MultiPlexer and Policy Executor to build the stream route map
Code Block | ||
---|---|---|
| ||
{ |
...
“policydef” : “…”
...
“policydef” : “…” “Metrics”: [“m1”, “m2”] |
...
}, |
...
{
“policydef” : “…”
...
{ “policydef” : “…” “Metrics”: [“m1”, “m3"] } ] |
}
...
Metric (Stream Schema):
Code Block |
---|
{
"name": "m1",
"sourceTopic" : "hadoopJmxMetric",
"fields": [
{name: metricname, type:string},
{name: timestamp, type:long},
{name: value, type:double}
]
} |
On boarding new metric flow
TBD