THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
In large clusters (500+ nodes), sometimes there are performance issues seen in AMS aggregations. In the ambari-metrics-collector log file, we can see log lines that look like
20:51:30,952 INFO 2080712366@qtp-974606690-381 AsyncProcess:1597 - #1, waiting for 13948 actions to finish
20:51:31,601 INFO 1279097595@qtp-974606690-359 AsyncProcess:1597 - #1, waiting for 19376 actions to finish
In Ambari 3.0.0, we are tackling these performance issues through a complete schema and aggregation logic revamp. Until then, we can use AMS whitelisting to reduce the number of metrics tracked by AMS, there by solving this scale problem.
How do we enable whitelisting in AMS.
Until Ambari 2.4.3
A metric whitelist file can be used to track the set of metrics in AMS. All other metrics will be discarded.
STEPS
- Metric whitelist file is present in /etc/ambari-metrics-collector/conf. If not present in older Ambari versions, it can be downloaded from https://github.com/apache/ambari/blob/trunk/ambari-metrics/ambari-metrics-timelineservice/conf/unix/metrics_whitelist to the collector host.
- Adding config ams-site : timeline.metrics.whitelist.file = <path_to_whitelist_file>
- Restart AMS collector
- Verify whitelisting config was used. In ambari-metrics-collector log file, verify the line 'Whitelisting # metrics'.
From Ambari 2.5.0 onwards
From Ambari 2.5.0, more refinements for whitelisting were included.
- App Blacklisting - Blacklist metrics from one or more services. Other service metrics will be entirely allowed or controlled through a whitelist file.
ams-site : timeline.metrics.apps.blacklist = hbase,namenode - App Whitelisting - Whitelist metrics from one or more services.
ams-site:timeline.metrics.apps.whitelist = nimbus,datanode NOTE : The App name can be found from the metadata URL - http:<metrics_collector_host>:6188/ws/v1/timeline/metrics/metadata
- Metric Whitelisting - Same as the whitelisting method in Ambari 2.4.3 (through a whitelist file).
These whitelisting/blacklisting techniques can be used together. For example,
- If you just have timeline.metrics.apps.blacklist = datanode, all datanode metrics will be disallowed. Metrics from all other services will be allowed.
- If you just have timeline.metrics.apps.whitelist = namenode, it is not useful since there is no blacklisting at all.
- If you have metric whitelisting enabled (through a file), and have timeline.metrics.apps.blacklist = datanode, all datanode metrics will be disallowed. The whitelisted metrics from other services will be allowed.
- If you have timeline.metrics.apps.blacklist = datanode, timeline.metrics.apps.whitelist = namenode and metric whitelisting enabled (through a file), datanode metrics will be blacklisted, all namenode metrics will be allowed, and whitelisted metrics from other services will be allowed.