...
Behavior | How to detect |
---|---|
High CPU usage | HBase process on Collector host taking up close to 100% of every core |
HBase Log: Compaction times | grep hbase-ams-master-<host>.log | grep "Finished memstore flush" This yields MB written in X milliseconds, generally 128 MBps and above is average speed unless the disk is contended. Also this search reveals how many times compaction ran per minute. A value greater than 6 or 8 is a warning that write volume are is far greater than what HBase can hold in memory |
HBase Log: ZK timeout | HBase crashes saying zookeeper session timed out. This happens because in embedded mode the zk session timeout is limited to max of 30 seconds (HBase issue: fix planned for 2.1.3). The cause is again slow disk reads. |
Collector Log : "waiting for some tasks to finish" | ambari-metric-collector log shows messages where AsyncProcess writes are queued up |
...
Upgrade to 2.1.2 is highly recommended.
Following is a list of fixes in 2.1.2 release that should greatly help to alleviate the slow loading and timeouts:
https://issues.apache.org/jira/browse/AMBARI-12654
...