Page History

Problem Statement

Eagle today does not have enough dynamic capabilities to support generic metric monitoring. Generic metric monitoring needs solve the following hard problems.

• Dynamically adding metrics in existing topology. It is common that one application may emit multiple metrics and the number of metrics may change. 
• Dynamically distributing metric and policy in existing topology. With multiple metrics and multiple policies per metric, even distribution of policy compute is critical
• Dynamically joining metrics. If one policy is for multiple metrics, we should make sure those multiple metrics come into this policy

Use Case Description

From Hadoop Native Metrics Monitoring, we can observed that 

...

This page's content is recorded in 

Design

Data Flow & Generic Topology Structure

Multi-Plexer

The key idea is to split the metric stream into different streams, so that it could be routed to different (logic) executor. This introduce a multiplexer. This multiplexer would be responsible to

...

So that given m1 as current tuple, it would multiplex routed to stream1 and stream2 by look at the route table.

Metadata

The current policy definition would be extended to express the relationship between policy and metric. This would be used in MultiPlexer and Policy Executor to build the stream route map

...

{
    "name": "m1",
    "sourceTopic" : "hadoopJmxMetric",
    "fields": [
       {name: metricname, type:string},
       {name: timestamp, type:long},
       {name: value, type:double}
    ]
}

On boarding new metric flow

TBD