Known Issues - 2.4.0

 The HBase normalizer which automatically splits / merges regions based on region size was leveraged in AMS (2.2.1). However, due to over aggressive region splitting by normalizer sometimes, in large clusters it could lead to an explosion of regions. This will eventually lead to AMS crashing every time it starts up.

As of 2.2.2, the AMS HBase normalizer cannot not be disabled through AMS configs.

Instructions for disabling normalizer on AMS HBase tables.

1. su ams (kinit if needed)
2. Connect to HBase shell on Metrics collector host.
/usr/lib/ams-hbase/bin/hbase --config /etc/ams-hbase/conf shell
3. Execute the following statements one by one.

 alter 'METRIC_RECORD', {NORMALIZATION_ENABLED => 'false'} 
 alter 'METRIC_AGGREGATE', {NORMALIZATION_ENABLED => 'false'}
 alter 'METRIC_RECORD_MINUTE', {NORMALIZATION_ENABLED => 'false'} 
 alter 'METRIC_AGGREGATE_MINUTE', {NORMALIZATION_ENABLED => 'false'}
 alter 'METRIC_RECORD_HOURLY', {NORMALIZATION_ENABLED => 'false'}
 alter 'METRIC_AGGREGATE_HOURLY', {NORMALIZATION_ENABLED => 'false'}
 alter 'METRIC_RECORD_DAILY', {NORMALIZATION_ENABLED => 'false'}
 alter 'METRIC_AGGREGATE_DAILY', {NORMALIZATION_ENABLED => 'false'}

4. Verify the configuration change took effect. Open the HBase master UI on a browser (http://<collector_host>:61310), and search for the String "NORMALIZATION". It should return no matches.

Ambari metrics collector shuts down and restarts randomly. The following error is seen in the collector log.

ERROR org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer: RECEIVED SIGNAL 15: SIGTERM

• Restart Ambari Server.
• Restart Ambari Metrics Monitor

On every host with a storm component [nimbus / supervisor / client] , carry out the following steps.

1. Verify broken link

   ls -al /usr/hdp/current/storm-<component>/lib/ambari-metrics-storm-sink.jar  

   Remove symlink

2. rm -f /usr/hdp/current/storm-<component>/lib/ambari-metrics-storm-sink.jar

   Reattach symlink to new JAR

3. ln -s /usr/lib/storm/lib/ambari-metrics-storm-sink*.jar 

   /usr/hdp/current/storm-<component>/lib/ambari-metrics-storm-sink.jar

Ambari Metrics service graphs might not show data for certain metrics and the the following error might be seen in the metrics collector log (Ambari 2.1.2, 2.1.2.1).

"The time range query for precision table exceeds row count limit, please query aggregate table instead"

• Change the ams-site configuration, Set timeline.metrics.service.default.result.limit = 15840
•  Restart the Collector

 Ambari Metrics Server wont start successfully with Kerberos in Distributed Mode (AMBARI-13711)

The problem is, we cannot have separate principals for HBase Master and RS. The Zookeeper ACLs will not allow znode created using one principal to be read by the other unless proper ACL are set.

Since master created the znode with a different principal than RS in 2.1.2 this will happen.

Change the AMS configuration to use the Master keytab and principal for RS
Set,

• ams-hbase-site ::: hbase.regionserver.keytab.file = /etc/security/keytabs/ams-hbase.master.keytab
• ams-hbase-site ::: hbase.regionserver.kerberos.principal = amshbasemaster/_HOST@REALM

Restart the Collector

- check /var/log/ambari-metrics-collector/ambari-metrics-collector.log for metrics data aggregation errors like OutOfOrderScannerNextException or SpoolTooBigToDiskException

- Set bigger value for hbase_regionserver_heapsize property in Advanced ams-hbase-env using Ambari Web UI

- Restart Metrics Collector.

AMBARI-11501,

AMBARI-12347

Issue 1:  (AMBARI-11501)
Steps to workaround this issue:

On the ambari server host

• cd /var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/params.py
• Edit hbase_staging_dir and point it to the desired location on HDFS, e.g.: "/ams-hbase/staging"
• Restart Ambari server.
• Restart Metrics Collector.

Issue 2: (AMBARI-12347)
Steps to workaround this issue:

• Edit the following properties in Advanced ams-hbase-security-site using Ambari Web UI:

ams.zookeeper.principal   =   zookeeper/_HOST@EXAMPLE.COM (Substitute appropriate REALM)

ams.zookeeper.keytab      =   /etc/security/keytabs/zk.service.keytab

Note: This is assuming you have a Zookeeper keytab on the host with Metrics Collector.

If not you should create one with appropriate permissions.

If a keytab already exists, make sure to chmod 440 /etc/security/keytabs/zk.service.keytab

Example: ]# klist -kt /etc/security/keytabs/zk.service.keytab

Keytab name: FILE:/etc/security/keytabs/zk.service.keytab
KVNO Timestamp         Principal
---- ----------------- --------------------------------------------------------
   1 07/08/15 22:29:07 zookeeper/ambari-sid-3.c.pramod-thangali.internal@EXAMPLE.COM

• Restart Metrics Collector

~]$ su - ams
~]$ export JAVA_HOME=/usr/jdk64/jdk1.8.0_40/
~]$ /usr/lib/ams-hbase/bin/hbase --config /etc/ams-hbase/conf shell

hbase(main):007:0> describe 'METRIC_RECORD'

------- HBase output describing table information ---------------

hbase(main):009:0> alter 'METRIC_RECORD', { NAME => '0', TTL => 172800}

hbase(main):007:0> describe 'METRIC_RECORD'

------- HBase output describing table information with new TTL ---------------

1. Copy core-site.xml and hdfs-site.xml to metric collector config directories after HA is enabled.

   cp /etc/hadoop/conf/core-site.xml /etc/ambari-metrics-collector/conf/ cp /etc/hadoop/conf/hdfs-site.xml /etc/ambari-metrics-collector/conf/ cp /etc/hadoop/conf/core-site.xml /etc/ams-hbase/conf/ cp /etc/hadoop/conf/hdfs-site.xml /etc/ams-hbase/conf/

2. Restart the Metrics Collector Component.