Problem Statement
Eagle today does not have enough dynamic capabilities to support generic metric monitoring. Generic metric monitoring needs solve the following hard problems.
- Dynamically adding metrics in existing topology. It is common that one application may emit multiple metrics and the number of metrics may change.
- Dynamically distributing metric and policy in existing topology. With multiple metrics and multiple policies per metric, even distribution of policy compute is critical
- Dynamically joining metrics. If one policy is for multiple metrics, we should make sure those multiple metrics come into this policy
Use Case Description
From Hadoop Native Metrics Monitoring, we can observed that
...
This page's content is recorded in
Jira | ||||||
---|---|---|---|---|---|---|
|
Design
Data Flow & Generic Topology Structure
Gliffy Diagram | ||||
---|---|---|---|---|
|
...
So that given m1 as current tuple, it would multiplex routed to stream1 and stream2 by look at the route table.
Metadata
The current policy definition would be extended to express the relationship between policy and metric. This would be used in MultiPlexer and Policy Executor to build the stream route map
...
Code Block |
---|
{ "name": "m1", "sourceTopic" : "hadoopJmxMetric", "fields": [ {name: metricname, type:string}, {name: timestamp, type:long}, {name: value, type:double} ] } |
On boarding new metric flow
TBD