...
Behavior | How to detect |
---|---|
High CPU usage | HBase process on Collector host taking up more than close to 100% of 1 every core |
HBase Log: Compaction times | grep hbase-ams-master-<host>.log | grep "Finished memstore flush" This yields MB written in X milliseconds, generally 128 MBps and above is average speed unless the disk is contended. Also this search reveals how many times compaction ran per minute. A value greater than 6 or 8 is a warning that write speeds are far greater than what HBase can hold in memory |
HBase Log: ZK timeout | HBase crashes saying zookeeper session timed out. This happens because in embedded mode the zk session timeout is limited to max of 30 seconds (HBase issue: fix planned for 2.1.3). The cause is again slow disk reads. |
Collector Log : "waiting for some tasks to finish" | ambari-metric-collector log shows messages where AsyncProcess writes are queued up |
...