Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

BehaviorHow to detect
High CPU usageHBase process on Collector host taking up close to 100% of every core
HBase Log: Compaction times

grep hbase-ams-master-<host>.log | grep "Finished memstore flush"

This yields MB written in X milliseconds, generally 128 MBps and above is average speed unless the disk is contended.

Also this search reveals how many times compaction ran per minute. A value greater than 6 or 8 is a warning that write volume are is far greater than what HBase can hold in memory

HBase Log: ZK timeout

HBase crashes saying zookeeper session timed out. This happens because in embedded mode the zk session timeout is limited to max of 30 seconds (HBase issue: fix planned for 2.1.3).

The cause is again slow disk reads.

Collector Log : "waiting for some tasks to finish"ambari-metric-collector log shows messages where AsyncProcess writes are queued up

...

Upgrade to 2.1.2 is highly recommended.

Following is a list of fixes in 2.1.2 release that should greatly help to alleviate the slow loading and timeouts:

https://issues.apache.org/jira/browse/AMBARI-12654

...