...
- User time clock
Sometimes we need user time clock to do statistics based on message time instead of system time, this is especially useful when analysis historical data - Tagged metric name
In eagle, tags is part of metric key(metricKey=metricName+tags+timestamp), which in dropwizard, metric key type is fixed as string, we need to do some encoding/decoding
So in order to solve the above problem, we extended dropwizard metric framework, the main change is using push mode to flush metric instead of pull mode to avoid contention issue(In dropwizard metric framework, it start a thread running metric reporter for metric status checking and flushing)
Following are some main class of eagle metric framework
| Code Block |
|---|
| language | java |
|---|
| title | IEagleMetric |
|---|
| collapse | true |
|---|
|
/**
* It's just a workaround to extends Gauge instead of Metric interface
* For MetricRegistry's notifyListenerOfRemovedMetric method will throw exception on unknown metric type
*/
public interface IEagleMetric extends Gauge<Double> {
void registerListener(EagleMetricListener listener);
/**
* return true if the metric need to be flushed, otherwise return false
* @param value
* @param userTimeClock
* @return
*/
boolean update(double value, long userTimeClock);
} |
...
| Code Block |
|---|
| language | java |
|---|
| title | EagleGaugeMetric |
|---|
| collapse | true |
|---|
|
public class EagleGaugeMetric extends EagleMetric {
private static final Logger LOG = LoggerFactory.getLogger(EagleGaugeMetric.class);
public EagleGaugeMetric(long latestUserTimeClock, String name, double value) {
super(latestUserTimeClock, name, value, 0);
}
public EagleGaugeMetric(EagleGaugeMetric metric) {
super(metric);
}
public boolean update(double d, long currentUserTimeClock) {
value.getAndSet(d);
this.latestUserTimeClock = currentUserTimeClock;
return true;
}
}
|
| Code Block |
|---|
| language | java |
|---|
| title | EagleMetricListener |
|---|
| collapse | true |
|---|
|
public interface EagleMetricListener {
/**
* The method should be called in thread-safe mode
* @param metrics
*/
void onMetricFlushed(List<EagleMetric> metrics);
} |
| Code Block |
|---|
| language | java |
|---|
| title | EagleServiceReporterMetricListener |
|---|
| collapse | true |
|---|
|
public class EagleServiceReporterMetricListener implements EagleMetricListener{
private EagleServiceClientImpl client;
private static final Logger LOG = LoggerFactory.getLogger(EagleServiceReporterMetricListener.class);
public EagleServiceReporterMetricListener(String host, int port, String username, String password) {
client = new EagleServiceClientImpl(host, port, username, password);
}
public EagleServiceReporterMetricListener(String host, int port) {
client = new EagleServiceClientImpl(host, port, null, null);
}
public void onMetricFlushed(List<EagleMetric> metrics) {
List<GenericMetricEntity> entities = new ArrayList<>();
for (EagleMetric metric : metrics) {
String metricName = metric.name;
entities.add(MetricEntityAdaptor.convert(metricName, metric));
}
try {
int total = entities.size();
GenericServiceAPIResponseEntity<String> response = client.create(entities, GenericMetricEntity.GENERIC_METRIC_SERVICE);
if(response.isSuccess()) {
LOG.info("Wrote " + total + " entities to service");
}else{
LOG.error("Failed to write " + total + " entities to service, due to server exception: "+ response.getException());
}
}
catch (Exception ex) {
LOG.error("Got exception while writing entities: ", ex);
}
}
} |
Eagle Metric List
- eagle.event.count: #event went into alert executor
- eagle.policy.eval.count: #policy evaluated times, which equals #policy * #event
- eagle.policy.eval.fail.count #event failed during evaluation
- eagle.alert.count: #alert generated
- eagle.alert.fail.count: #alert not successfully generated
- eagle.kafka.message.consumer.lag: lag between total kafka offset & comsuer's offset
- eagle.kafka.message.count: kafka message count on a specific key, like user=hadoop
