If you follow the installation instructions, Trafodion should start up correctly. However, since each system has a somewhat different setup, you may encounter problems during or after an install. This page talks about how Trafodion objects are set up, what CLASSPATHs and objects are required, and related topics. It also lists common installation problems and their solutions. If you need to solve such issues or if these topics interest you, please read on.
To configure max client connections in Zookeeper on a Hortonworks cluster, do the following:
Note: On a Cloudera cluster, the installer performs these actions automatically. (This will soon be true for Hortonworks as well.)
Before even looking at the Trafodion executables, you should make sure that HBase is set up correctly to work with it. Trafodion introduces new code to HBase in form of a jar containing HBase-trx. This jar is copied into the HBase lib directory during the installation process, or sometimes you might need to do this manually. Some things to consider:
ClassNotFoundException:TransactionalRegionServer.The Trafodion installer also adds several properties to the HBase configuration file, hbase-site.xml. For example:
<property>
<name>hbase.hregion.impl</name>
<value>org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion</value>
</property>
<property>
<name>hbase.coprocessor.region.classes</name>
<value>org.apache.hadoop.hbase.coprocessor.transactional.TrxRegionObserver,
org.apache.hadoop.hbase.coprocessor.transactional.TrxRegionEndpoint,
org.apache.hadoop.hbase.coprocessor.AggregateImplementation
</value>
</property>Make sure you see an entry to hbase.hregion.impl, pointing to org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion, and a coprocessororg.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion. Those are just two examples. If they are missing, something went wrong with the installation.
For Trafodion SQL aggregation, the org.apache.hadoop.hbase.coprocessor.AggregateImplementation coprocessor is used for non-HBase-trx transactional queries. Aggregation routines in theorg.apache.hadoop.hbase.coprocessor.transactional.TrxRegionEndpoint coprocessor are used for HBase-trx transactional queries.
Once you feel that HBase should work correctly, validate this with the hbase shell, by using some simple commands like the following:
sudo su - trafodion hbase shell create 'testtab','cf1' put 'testtab','row1','cf1:col1','val1' scan 'testtab' disable 'testtab' drop 'testtab' quit
Trafodion runs under a user ID, trafodion, that has been created on your system by the Trafodion installer. The trafodion user ID needs to have passwordless ssh access to all nodes in the cluster. The installer should have created public and private keys in ~trafodion/.ssh to set this up. During setup, you should be able to su (or sudo su) to this user ID. Once things are set up, you probably won't want to use this ID anymore. It should definitely have a strong password, or, better, no password at all.
The installer adds commands to ~trafodion/.bashrc to set up some environment variables. Most of this setup is done through files $MY_SQROOT/sqenv.sh and $MY_SQROOT/sqenvcom.sh. MY_SQROOT is set up in .bashrc. Make some basic checks:
trafodion user ID is able to ssh to all the nodes listed in $MY_NODES, without supplying a password.sudo su - trafodion echo $MY_SQROOT echo $DCS_INSTALL_DIR pdsh $MY_NODES pwd echo $CLASSPATH
The class path is probably the main place where issues with Trafodion installation surface. It's a long CLASSPATH, consisting of the following parts:
If you see elements of this CLASSPATH missing (often indicated by consecutive colons in the middle of the CLASSPATH), take a look at the sqenvcom.sh file and how it sets up the CLASSPATH. Often there is a simple fix to the problem. Note that the configuration directories are required to access HBase as well as to get to Hive metadata.
Typical issues with the class path:
Fixing class path issues may require a few extra steps. Let's say you fixed this by making a small patch to the $MY_SQROOT/sqenvcom.sh file, or by copying some jar /somedir/newjar.jar. Since Trafodion runs distributed on the cluster, it requires another step to propagate the patched environment to all nodes. There is a utility, called sqgen, that creates needed files to set up the CLASSPATH and other information and copies some generated files across the cluster. Here is an example of what you might do as the trafodion user. Please modify these commands as applicable to you:
sudo su - trafodion # stop Trafodion sqstop cd $MY_SQROOT # patch environment, for example in this file vi sqenvcom.sh # now use the pdcp command to copy the patched file to all nodes # The $MY_NODES variable is handy to do this. pdcp $MY_NODES sqenvcom.sh $PWD # test the patch on all nodes by checking the classpath pdsh $MY_NODES echo $CLASSPATH | grep /usr/lib/hbase/hbase-trx | wc -l # copy other needed files pdcp $MY_NODES /somedir/newjar.jar /somedir # secure the new file pdsh $MY_NODES chmod 644 /somedir/newjar.jar # validate that the HBase-trx jars are in the right place (see HBase setup topic above) # note that we put the name with wildcards in quotes, so it is resolved on the target system pdsh $MY_NODES ls -l "/usr/lib/hbase/lib/*trx*" # use a new shell (important) to generate a new environment for Trafodion ssh localhost sqgen exit # exit all shells with the old environment and restart Trafodion exit sudo su - trafodion sqstart
Trafodion uses a Java interface to access HBase, and the native libhdfs library to access the HDFS files of Hive tables. If you have set permissions for HBase and HDFS, they must permit the trafodion user ID to access the necessary HBase tables and HDFS files. The easiest way to test this is to run HBase, Hive and HDFS commands as the trafodion user. See above for simple test that can be done in the HBase shell.
Again, due to the fact that Trafodion manages a cluster, there are a few special things to know and check. Trafodion has its own set of processes. As the trafodion user, you can use the sqps command to display all the Trafodion processes running on the cluster. Example (simplified):
sudo su - trafodion $ sqps ... [$Z000LSB] NID,PID(os) PRI TYPE STATES NAME PARENT PROGRAM [$Z000LSB] ------------ --- ---- ------- ----------- ----------- --------------- [$Z000LSB] 000,00022519 001 DTM ES--A-- $TM0 NONE tm [$Z000LSB] 000,00024392 001 GEN ES--A-- $Z000JWX NONE mxosrvr [$Z000LSB] 000,00024441 001 GEN ES--A-- $Z000JYB NONE mxosrvr [$Z000LSB] 000,00005348 001 GEN ES--A-- $Z0004CT $Z000JWX tdm_arkcmp [$Z000LSB] 000,00013085 001 GEN ES--A-- $Z000ANV $Z0004CT tdm_arkcmp [$Z000LSB] 001,00022612 001 DTM ES--A-- $TM1 NONE tm ...
NID,PID show the node number and process ID. To translate the numeric node ID into a node name, use this command:
sqshell -c node
What do these processes do?
For more details, see Architecture. We are only concerned with troubleshooting problems on this page.
When a client connects through JDBC/ODBC, it first contacts a master process on one of the cluster nodes and obtains the IP address and port of an available mxosrvr, anywhere on the Trafodion cluster. To be able to connect, confirm that this port has been opened in the firewall. By default, the range of ports starts at 37800, and the size of the range depends on how many mxosrvrs are configured.
Another potential problem: A node may have multiple IP addresses, for example an internal network address and an external address. Since JDBC/ODBC connection requests get redirected to another node, DCS (Trafodion Database Connectivity Services) needs to give the correct IP address to the client. This is configured in the $DCS_INSTALL_DIR/conf/dcs-site.xml file, using property dcs.dns.interface.
A few things to consider:
trafodion user, run the hive shell and try some simple commandssudo su - trafodion hive show tables; select count(*) from <some table>; quit;
Trafodion implements the ability to recover transactions after an HBase crash. This feature is enabled by default.
Other scenarios, such as recovery from failure of individual RegionServers or the failure of the TM process, are not yet supported.
If you would like to disable recovery, set the following switches in the ms.env file and restart the Trafodion instance.
TM_ENABLE_TLOG_WRITES=0
Property hbase.regionserver.region.transactional.tlog: This property controls the behavior of the regions during recovery.
<property>
<name>hbase.regionserver.region.transactional.tlog</name>
<value>true</value>
</property>
This property is currently set to true.
When a Trafodion table is created, there will be configuration objects created in both Trafodion and HBase. There may be times where these two sets of configuration objects become inconsistent. This can result in the inability to drop and recreate the table for use in Trafodion. You might see Trafodion errors trying to drop objects.
SQL>drop table trafodion.seabase.table1; *** ERROR[1389] Object TRAFODION.SEABASE.TABLE1 does not exist in Trafodion. [2014-06-10 14:12:51]
To recover from this problem, you need to ensure that all Trafodion and HBase configuration objects for the table have been removed. To remove a table from HBase, use the HBase shell to disable and drop the table.
hbase shell disable 'TRAFODION.SEABASE.TABLE1'; drop 'TRAFODION.SEABASE.TABLE1';
You may also need to ensure all Trafodion metadata has been removed. You will query the Trafodion metadata to obtain the table's object_uid. Use the object_uid returned to the select query in the delete SQL statements as a replacement for <num>.
trafci select * from "_MD_".objects where object_name = 'TABLE1' and schema_name = 'SEABASE' and object_type = 'BT';
If a row is not returned, then the table configuration is not present in the Trafodion metadata. You can exit from "trafci". If a row is returned, use the value returned for the 'object_uid' column in the following delete statements. This will remove all metadata configuration information for the table.
begin work; set parserflags 131072; delete from "_MD_".objects where object_uid = <num>; delete from "_MD_".columns where object_uid = <num>; delete from "_MD_".keys where object_uid = <num>; delete from "_MD_".tables where table_uid = <num>; delete from "_MD_".table_constraints where table_uid = <num>; If all is well at this point, then… commit work; exit;
There may be times when you would like to refresh the Trafodion instance and matching HBase objects. NOTE: This will destroy all tables that were created in Trafodion! You can perform a series of steps that will stop the Trafodion instance and then clean out the HBase and ZooKeeper objects. You may then restart the Trafodion instance and reinitialize the Trafodion objects.
sudo su - trafodion # stop Trafodion sqstop
hbase shell disable_all 'TRAFODION.*' drop_all 'TRAFODION.*' exit
/usr/lib/zookeeper/bin/zkCli.sh ls2 /hbase/table rmr /hbase/table/TRAFODION.<schema>.<table>
trafodion user ID does not have sudo privilege.sudo su hdfs --command "hadoop dfs -ls /hbase"
sudo su hdfs –command "hadoop dfs –rm –r /hbase/TRAFODION.CMP*"
sudo su hbase --command "hbase hbck -repair"
sudo su - trafodion # start Trafodion sqstart
sudo su - trafodion sqlci | cat initialize trafodion; exit;
Online node reintegration integrates an offline Trafodion node back into the cluster or instance for a previously failed (HWR or SWR) Trafodion node without taking the instance down. HWR node failures require that the node be repaired or replaced and then placed back into the original physical rack location. SWR node failures may be due to failure of the underlying Linux OS or due to fail-safe protection triggered by Trafodion. After the node is healthy and functional, ensure that the HBase environment on the node is also available. The final step is to reintegrate the node into the Trafodion instance. Currently, reintegration is a manually initiated operation.
In the following example, node ”sea-dev4” is being reintegrated into the Trafodion instance:
trafodion@sea-dev1 ~]$ sqshell -a Processing cluster.conf on local host sea-dev1 [$Z000HAT] Shell/shell Version 1.0.1 Release 1.0.0 (Build debug [1.0.0rc2-6-ge7cad10_Bld72], date 20150125_083000) [$Z000HAT] %node info [$Z000HAT] Logical Nodes = 6 [$Z000HAT] Physical Nodes = 6 [$Z000HAT] Spare Nodes = 0 [$Z000HAT] Available Spares = 0 [$Z000HAT] NID Type State Processors #Procs [$Z000HAT] PNID State #Cores MemFree SwapFree CacheFree Name [$Z000HAT] --- ----------- -------- ---------- -------- -------- --------- -------- [$Z000HAT] 000 Any Up 2 9 [$Z000HAT] 000 Up 8 6517656 4095996 10228088 sea-dev1 [$Z000HAT] 001 Any Up 2 10 [$Z000HAT] 001 Up 8 6308828 4095992 11607900 sea-dev2 [$Z000HAT] 002 Any Up 2 8 [$Z000HAT] 002 Up 8 1988592 3982764 11939032 sea-dev3 [$Z000HAT] 003 Any Down [$Z000HAT] 003 Down sea-dev4 [$Z000HAT] 004 Any Up 2 8 [$Z000HAT] 004 Up 8 2081148 3832532 10454704 sea-dev5 [$Z000HAT] 005 Any Up 2 8 [$Z000HAT] 005 Up 8 905744 3836724 10166456 sea-dev6 [$Z000HAT] %up sea-dev4 [$Z000HAT] 01/27/2015-15:58:48 - Node sea-dev4 is merging to existing cluster. [$Z000HAT] 01/27/2015-15:58:49 - Node sea-dev4 join phase starting. [$Z000HAT] %[$Z000HAT] 01/27/2015-15:58:54 - Node 3 (sea-dev4) is UP [$Z000HAT] %[$Z000HAT] %node info [$Z000HAT] Logical Nodes = 6 [$Z000HAT] Physical Nodes = 6 [$Z000HAT] Spare Nodes = 0 [$Z000HAT] Available Spares = 0 [$Z000HAT] NID Type State Processors #Procs [$Z000HAT] PNID State #Cores MemFree SwapFree CacheFree Name [$Z000HAT] --- ----------- -------- ---------- -------- -------- --------- -------- [$Z000HAT] 000 Any Up 2 9 [$Z000HAT] 000 Up 8 6540244 4095996 10251084 sea-dev1 [$Z000HAT] 001 Any Up 2 10 [$Z000HAT] 001 Up 8 6287700 4095992 11587408 sea-dev2 [$Z000HAT] 002 Any Up 2 8 [$Z000HAT] 002 Up 8 2180728 3982764 11888216 sea-dev3 [$Z000HAT] 003 Any Up 2 7 [$Z000HAT] 003 Up 8 1731684 3495860 11720212 sea-dev4 [$Z000HAT] 004 Any Up 2 8 [$Z000HAT] 004 Up 8 2257312 3832532 10465852 sea-dev5 [$Z000HAT] 005 Any Up 2 8 [$Z000HAT] 005 Up 8 1167184 3836724 10187724 sea-dev6 [$Z000HAT] %
NOTE: If you do not see a problem listed here, see the known problems for the Trafodion release that you are using:
To troubleshoot problems related to data loading, see Troubleshooting Data Loading Problems.
Symptom: When trying to start Trafodion, you get the following loop:
# of Transactions being recovered: 0 Checking..... SQ registry key SQ_TXNSVC_READY is not set # of Transactions being recovered: 0 Checking..... SQ registry key SQ_TXNSVC_READY is not set # of Transactions being recovered: 0 Checking.....
Cause: DTM is not starting.
Solution: Common reasons and solutions:
Possible reasons:The $MY_SQROOT/export/lib/ hbase-trx jar file that got copied to /usr/lib/hbase/lib does not exist or does not contain the correct permissions, which should be 644.The Hadoop Java version is different from what shows with the java -version command. To determine the Hadoop Java version, use top and then press C and look for the Java processes and their version. In Trafodion Release 0.8.n, the Hadoop Java version is either 1.6 or 1.7. Starting in Trafodion Release 0.9.0, the minimum Java 7 version required is 1.7.0_65. In Trafodion Release 1.1.0, the minimum Java 7 version required is 1.7.0_67. To force Hadoop to use the right version, run this command: yum -y remove.
Symptoms:
SQL>initialize trafodion
+>;
*** ERROR[8448] Unable to access Hbase interface. Call to ExpHbaseInterface::checkAndInsertRow returned error HBASE_ACCESS_ERROR(-706). Cause:
java.io.IOException: Problem with calling coprocessor, no regions returned result
org.apache.hadoop.hbase.client.transactional.TransactionalTable.checkAndPut(TransactionalTable.java:500)
org.apache.hadoop.hbase.client.transactional.RMInterface.checkAndPut(RMInterface.java:371)
org.trafodion.sql.HBaseAccess.HTableClient.putRow(HTableClient.java:1023)
org.trafodion.sql.HBaseAccess.HBaseClient.insertRow(HBaseClient.java:1490)
. [2015-09-23 23:31:53] *** ERROR[8839] Transaction was aborted. [2015-09-23 23:31:53]
Cause: There can be multiple potential issues for either symptom
Solution:
Symptom: While running update statistics or count(*) commands, you see org.apache.hadoop.hbase.client.ScannerTimeoutException errors.
Cause: The default hbase.rpc.timeout and hbase.client.scanner.timeout.period values might be too low given the size of the tables.
Solution: Increase the hbase.rpc.timeout and hbase.client.scanner.timeout.period values. We have found that increasing those values to 600 seconds (10 minutes) will prevent many timeout-related errors. For more information, see the HBase Configuration and Fine Tuning Recommendations.
Symptom: SQL queries fail with HBase OutOfOrderScannerNextException and WrongRegionException errors due to HBase regions being moved during region load balancing.
Cause: If HBase region load balancing is enabled for your installation and if regions are moved with active DTM transactions, there can be unpredictable results, as transactional context can be lost.
Solution: To avoid this problem, use the HBase shell command balance_switch false to disable the movement of a region from one server to another. For example:
# hbase shell hbase(main):002:0> balance_switch false true <-- Output will be the last setting of the balance_switch value 0 row(s) in 0.0080 seconds
Symptom: You see an ssh_exchange error when starting many client connections to the Trafodion system.
Cause: The settings for maximum sessions and maximum startups in OpenSSH are too low.
Solution: Modify the /etc/ssh/sshd_config file with these settings:
MaxSessions 100 MaxStartups 200
Then, restart the sshd service.