Introduction

Application "HADOOP_JMX_METRIC_MONITOR" provide embedded collector script to ingest hadoop/hbase jmx metric as eagle stream and provide ability to define alert policy and detect anomaly in real-time from metric.

Setup & Installation

1. Make sure already setup a site (here use a demo site named "sandbox")
2. Install "Hadoop JMX Monitor" app in eagle server
   
3. Configure Application settings
   
4. Ensure a kafka topic named hadoop_jmx_metric_{SITE_ID} (In current guide, it should be hadoop_jmx_metric_sandbox)
5. Setup metric collector for monitored Hadoop/HBase using hadoop_jmx_collector and modify the configuration
   1. Collector scripts: https://github.com/apache/incubator-eagle/tree/master/eagle-external/hadoop_jmx_collector

   2. Rename config-sample.json to config.json: https://github.com/apache/incubator-eagle/blob/master/eagle-external/hadoop_jmx_collector/config-sample.json

      config.json

      {
      	env: {
      		site: "sandbox",
      		name_node: {
      			hosts: [
      				"sandbox.hortonworks.com"
      			],
      			port: 50070,
      			https: false
      		},
      		resource_manager: {
      			hosts: [
      				"sandbox.hortonworks.com"
      			],
      			port: 50030,
      			https: false
      		}
      	},
      	inputs: [{
      		component: "namenode",
      		host: "server.eagle.apache.org",
      		port: "50070",
      		https: false,
      		kafka_topic: "nn_jmx_metric_sandbox"
      	}, {
      		component: "resourcemanager",
      		host: "server.eagle.apache.org",
      		port: "8088",
      		https: false,
      		kafka_topic: "rm_jmx_metric_sandbox"
      	}, {
      		component: "datanode",
      		host: "server.eagle.apache.org",
      		port: "50075",
      		https: false,
      		kafka_topic: "dn_jmx_metric_sandbox"
      	}],
      	filter: {
      		monitoring.group.selected: [
      			"hadoop",
      			"java.lang"
      		]
      	},
      	output: {
      		kafka: {
      			brokerList: [
      				"localhost:9092"
      			]
      		}
      	}
      }

6. Click "Install" button then you will see the HADOOP_JMX_METRIC_STREAM_{SITE_ID} in Streams
   

Usage

Define JMX Alert Policy

1. Go to "Define Policy"
2. Select HADOOP_JMX_METRIC_MONITOR related streams

3. Define SQL-Like policy, for example 

   from HADOOP_JMX_METRIC_STREAM_SANDBOX[metric=="cpu.usage" and value > 0.9] 
   select site,host,component,value 
   insert into HADOOP_CPU_USAGE_GT_90_ALERT;