THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
Description
From Hadoop Native Metrics Monitoring, we can observed that
- A typical metric stream should contain multiple metrics with different names. More often, these metrics are grouped in one semantic category, say JMX metric, system metric.
- Multiple monitoring rules are required based on different metric name/type.
- CEP logic becomes more and more complex when user needs to do stream separation and join(pattern match).
- Un-named stream make the data flow logically route from spout to next elements. Though the grouping helps here, with named stream, we could make route for the different metrics.
This page's content is recorded in
Design
Data Flow & Generic Topology Structure
Multi Plexer
The key idea is to split the metric stream into different streams, so that it could be routed to different (logic) executor. This introduce a multiplexer. This multiplexer would be responsible to
- Understand the metric by metadata driven (a "metricName" field in the input tuple)
- Multiplex the metrics into different stream names (This is driven the metric name/type)
Policy Executor
Play a role similar to the current PolicyProcessExecutor, it would recognize the metric in the same way of MultiPlexer. And support stream joining.
Route Table
A route table is calculated and used by Multiplexer to decide how to route the given tuple. Basically, the route table ask the question "Given current tuple, which stream should the multiplexer route it to."
It would be implemented like
| MetricName | Policy | Stream |
|---|---|---|
| m1 | p1 | stream1 |
| m1 | p2 | stream2 |
| m1 | p3 | stream1 |
| m2 | p2 | stream2 |
So that given m1 as current tuple, it would multiplex routed to stream1 and stream2 by look at the route table.
Metadata
The current policy definition would be extend to express the relationship between policy and metric. This would be used in MultiPlexer and Policy Executor to build the stream route map
{
“policydef” : “…”
“Metrics”: [“m1”, “m2”] — 映射到#2的metric name
},
{
“policydef” : “…”
“Metrics”: [“m1”, “m3"]
}
]
On boarding
TBD