You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 21 Next »

Suggested Memory settings

Cluster SizeRecommended Mode

Collector Heapsize

ams-env : metrics_collector_heapsize

HBase Master Heapsize

ams-hbase-env : hbase_master_heapsize

HBase RS Heapsize

ams-hbase-env : hbase_regionserver_heapsize

HBase Master xmn size

ams-hbase-env : hbase_master_xmn_size

HBase RS xmn size

ams-hbase-env : regionserver_xmn_size

1 - 10Embedded5121408512192-
11 - 20Embedded10241920512256-
21 - 100Embedded16645120512768-

100 - 300

Embedded4352130565122048-
300 - 500Distributed4352512130561022048
500 - 800Distributed7040512211201023072
800 - 1000Distributed11008512327681025120
1000+Distributed13696512327681025120

Identifying and tackling scale problems in AMS through configs

How do we find out if AMS is experiencing scale problems?

One or more of the following consequences can be seen on the cluster.

  • Metrics Collector shuts down intermittently. Since Auto Restart is enabled for Metrics collector by default, this will up show as an alert stating 'Metrics collector has been auto restarted # times the last 1 hour'.
  • Partial data is seen.
    • All non-aggregated host metrics are seen (HDFS Namenode metrics  / Host summary page on Ambari / System - Servers Grafana dashboard).
    • Aggregated data is not seen. (AMS Summary page / System - Home Grafana dashboard / HBase - Home Grafana dashboard).

Systematically Troubleshooting the scale issue

  • Get the current state of the system
What to get?How to get?Is there a Red flag? 
How long does it take for 2 min aggregator to finish?grep "TimelineMetricClusterAggregatorSecond" /var/log/ambari-metrics-collector/ambari-metrics-collector.log?  
How many metrics are being collected?
  • curl -K http://<ams-host>:6188/ws/v1/timeline/metrics/metadata -o /tmp/metrics_metadata.txt
  • Number of metrics is the output of the command 'grep -o "metricname" /tmp/metrics_metadata.txt | wc -l'

>15000 could be a problem. Find the component contributing maximum to the number of metrics d

Find the component contributing maximum to the numbe
What is the number of regions and store files in AMS HBase?

This can be got from AMS HBase Master UI.

http://<METRICS_COLLECTOR_HOST>:61310

  
    
    
    




Advanced Configurations

ConfigurationPropertyDescriptionMinimum Recommended values (Host Count => MB)
ams-sitephoenix.query.maxGlobalMemoryPercentage

Percentage of total heap memory used by Phoenix

threads in the Metrics Collector API/Aggregator daemon.

20 - 30, based on available memory. Default = 25.
ams-sitephoenix.spool.directorySet directory for Phoenix spill files. (Client side)Set this to different disk from hbase.rootdir dir if possible.
ams-hbase-sitephoenix.spool.directorySet directory for Phoenix spill files. (Server side)Set this to different disk from hbase.rootdir dir if possible.
ams-hbase-sitephoenix.query.spoolThresholdBytes

Threshold size in bytes after which results from parallelly

executed query results are spooled to disk.

Set this to higher value based on available memory.

Default is 12 mb.

  • No labels