...
Note | ||
---|---|---|
| ||
You MUST complete the steps in Create Build Environment and build the Trafodion components using the instructions in Build Source before you create the test environment. |
Prerequisites
The following prerequisites need to be met in order to create a functional Trafodion Test Environment.
Passwordless ssh
Do the following to check whether you have passwordless SSH setup correctly.
...
If the ssh localhost
command prompts for a password, then passwordless ssh
is not set up correctly. The following examples set up passwordless ssh
using id_rsa
keys. You can choose the method that best represents your environment.
If you already have an existing set of ssh keys
Simply copy both the id_rsa.pub
and id_rsa
to your ~/.ssh
directory.c. Then, do the following to modify your ssh
environment.
Code Block | ||||
---|---|---|---|---|
| ||||
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys chmod 600 ~/.ssh/id_rsa echo "NoHostAuthenticationForLocalhost=yes" >>~/.ssh/config chmod go-w ~/.ssh/config chmod 755 ~/.ssh; chmod 640 ~/.ssh/authorized_keys; cd ~/.ssh; chmod 700 .. |
If you need to create your keys first
Do the following:
Code Block | ||||
---|---|---|---|---|
| ||||
rm -rf ~/.ssh ssh-keygen -t rsa -N "" -f ~/.ssh/id_rsa cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys chmod 600 ~/.ssh/id_rsa.pub echo "NoHostAuthenticationForLocalhost=yes" >>~/.ssh/config chmod go-w ~/.ssh/config chmod 755 ~/.ssh; chmod 640 ~/.ssh/authorized_keys; cd ~/.ssh; chmod 700 .. |
System Limits
Please check that the system limits in your environment are appropriate for Apache Trafodion. If they are not, then you will need to increase the limits or Trafodion cannot start.
Use the following command to check your system limits:
Code Block | ||
---|---|---|
| ||
ulimit -a |
The recommended settings are as follows:
Code Block | ||||
---|---|---|---|---|
| ||||
core file size (blocks, -c) 1000000 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 515196 max locked memory (kbytes, -l) 49595556 max memory size (kbytes, -m) unlimited open files (-n) 32000 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 10240 cpu time (seconds, -t) unlimited max user processes (-u) 267263 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited |
Please refer to http://www.computerworld.com/article/2693414/setting-limits-with-ulimit.html for information on how you change system limits.
Verify Network Setup
Hadoop services typically require a functional resolution fully-qualified network domain name (FQDN) to function properly. The FQDN network configuration is located in /etc/hosts
while name resolution is configured using /etc/resolv.conf
.
Find FQDN
The hostname
utility allows you to find the short name as well as the FQDN for your server.
Code Block | ||||
---|---|---|---|---|
| ||||
$ # Get current hostname $ hostname box01 $ # Get fully-qualified domain name for server $ hostname --fqdn box01.trafodion.org $ |
Test FQDN
The ssh
utility allows you to test whether the configured correctly.
Code Block | ||||
---|---|---|---|---|
| ||||
$ ssh box01.trafodion.org # Command hangs, ctrl-c to break $ |
Modify FQDN Resolution
Info | ||
---|---|---|
| ||
Some of the steps in this subsection requires root access to change configurations and run utilities. |
Name resolution is performed per the configuration in /etc/resolv.conf
while /etc/hosts
configures the FQDN.
Code Block | ||||
---|---|---|---|---|
| ||||
$ # Check FQDN resolution (fail example) $ host -T Host box01.trafodion.org not found: 5(REFUSED) # Examine issue $ cat /etc/resolv.conf # Generated by NetworkManager nameserver 192.168.1.1 $ cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 10.0.2.15 box01.trafodion.org box01 |
In this example, the name server is in a different domain than box01.trafodion.org.
Code Block | ||||
---|---|---|---|---|
| ||||
$ # Check interface configuration $ sudo ifconfig eth3 Link encap:Ethernet HWaddr 08:00:27:D0:B7:40 inet6 addr: fe80::a00:27ff:fed0:b740/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:1703 errors:0 dropped:0 overruns:0 frame:0 TX packets:3778 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:195684 (191.0 KiB) TX bytes:1290492 (1.2 MiB) eth4 Link encap:Ethernet HWaddr 08:00:27:A5:29:28 inet addr:192.168.1.12 Bcast:192.168.1.255 Mask:255.255.255.0 inet6 addr: fe80::a00:27ff:fea5:2928/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:282605 errors:0 dropped:0 overruns:0 frame:0 TX packets:148931 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:290384816 (276.9 MiB) TX bytes:18066069 (17.2 MiB) lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:263 errors:0 dropped:0 overruns:0 frame:0 TX packets:263 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:41810 (40.8 KiB) TX bytes:41810 (40.8 KiB) |
The server is running on an192.168.1.x
network. Therefore, this example shows an issue with the/etc/hosts
configuration. Correct the setting as root editing the /etc/hosts. In this example, the eth4 address is 192.168.1.12 yielding the following result.
Code Block | ||||
---|---|---|---|---|
| ||||
$ cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.1.12 box01.trafodion.org box01 $ ssh box01.trafodion.org The authenticity of host 'box01.trafodion.org (192.168.1.12)' can't be established. RSA key fingerprint is 88:f7:90:fe:35:76:10:92:cf:6d:c4:60:ca:b9:43:9f. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'box01.trafodion.org,192.168.1.12' (RSA) to the list of known hosts. Last login: Tue Jan 26 13:41:58 2016 from localhost $ exit $ ssh box01 The authenticity of host 'box01 (192.168.1.12)' can't be established. RSA key fingerprint is 88:f7:90:fe:35:76:10:92:cf:6d:c4:60:ca:b9:43:9f. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'box01' (RSA) to the list of known hosts. Last login: Tue Jan 26 14:07:09 2016 from box01.trafodion.org $ exit $ |
Repeat for each server in your environment to ensure proper FQDN resolution before installing Hadoop.
Install Hadoop and Trafodion
You can create a Trafodion test environment using a:
- Pre-Installed Hadoop: Trafodion installation on a system that already has a compatible version of Hadoop installed.
- Local Hadoop: You install a Hadoop environment using the
install_local_hadoop
script
Your installation approach depends on whether you already have installed Hadoop.
Anchor | ||||
---|---|---|---|---|
|
Pre-Installed Hadoop
Info | ||
---|---|---|
| ||
Currently, Trafodion requires:
|
Build Binary tar Files
Build the Trafodion binary tar files.
Code Block | ||||
---|---|---|---|---|
| ||||
cd <Trafodion source directory> make package |
For more information, refer to Build Trafodion Components in the Build Source chapter.
Install Trafodion
To be written.
Anchor | ||||
---|---|---|---|---|
|
Local Hadoop
Use the following instructions to install a local Hadoop environment.
Run install_local_hadoop
The install_local_hadoop
script downloads compatible versions of Hadoop, HBase, Hive, and MySQL based on the Cloudera archives. Then, it starts Trafodion.
Info | ||
---|---|---|
| ||
|
Command | Usage |
---|---|
install_local_hadoop | Uses default ports for all services. |
install_local_hadoop -p fromDisplay | Start Hadoop with a port number range determined from the DISPLAY environment variable. |
install_local_hadoop -p rand | Start with any random port number range between 9000 and 49000. |
install_local_hadoop -p <port> | Start with the specified port number. |
Sample Procedure
Start a new
ssh
session and ensure that the Trafodion environmental variables are loaded.Code Block language bash cd <Trafodion source directory> source ./env.sh
Code Block language bash title Example: Load Environment Variables cd mysource/incubator-trafodion source ./env.sh
Install the Hadoop software.
Code Block language bash cd $MY_SQROOT/sql/scripts install_local_hadoop ./install_traf_components
Code Block language bash title Example: Install Hadoop Software $ cd $MY_SQROOT/sql/scripts $ install_local_hadoop Checking for existing Hadoop processes... The testware tpcds_kit.zip does not exist and will not be installed This testware is needed to run developer HIVE regression tests . . . Installed directory size and name = 2.7G /home/trafdeveloper/mysource/incubator-trafodion/core/sqf/sql/local_hadoop Setup is complete. You can use the convenience scripts starting with sw... located in /home/trafdeveloper/mysource/incubator-trafodion/core/sqf/sql/scripts. $ ./install_traf_components Installing and configuring DCS, REST, TRAFCI & Phoenix tests for Trafodion... Environment used for core, DCS, REST and Phonenix ... . . . Configuration scripts for DCS, REST, TRAFCI and Phoenix test are set up Open a new session and start Trafodion by executing sqgen and sqstart scripts $
Verify installation.
Code Block language bash $ swstatus 6 java servers and 2 mysqld processes are running 713 NameNode 19513 HMaster 1003 SecondaryNameNode 838 DataNode 1173 ResourceManager 1298 NodeManager
The following Java servers should be running:
NodeManager (Yarn)
ResourceManager (Yarn)
NameNode (HDFS)
SecondaryNameNode (HDFS)
DataNode (HDFS)
HMaster (HBase)
In addition, 2 mysqld processes should be running.
Info The Hadoop enviroment is install in:
$MY_SQROOT/sql/local_hadoop
. The log files are located in$MY_SQROOT/sql/local_hadoop/log
and in the Hadoop components' log directories. If a service is not started, then look in the associated log files for the issue and search for the error string for possible solutions to the issue; these are Hadoop issues and are therefore outside the scope of this guide.
- Run sqgen —
sqgen
creates the Trafodion configuration files. Do the following:Start a new terminal window for this step.
Do the following:
Code Block language bash cd <Trafodion source directory> source ./env.sh cd $MY_SQROOT/etc # delete ms.env, if it exists rm ms.env cd $MY_SQROOT/sql/scripts sqgen
Code Block language bash title Example: Run sqgen $ cd mysource/incubator-trafidion $ source ./env.sh $ cd $MY_SQROOT/etc # delete ms.env, if it exists $ rm ms.env $ cd $MY_SQROOT/sql/scripts $ sqgen Workstation environment - Not a clustered environment Generating SQ environment variable file: /home/trafdeveloper/mysource/incubator-trafodion/core/sqf/etc/ms.env Note: Using cluster.conf format type 2. . . . *********************************************************** Updating Authentication Configuration *********************************************************** Creating folders for storing certificates $
Start Trafodion.
Code Block language bash title Example: sqstart $ sqstart Checking orphan processes. Removing old mpijob* files from /home/trafdeveloper/mysource/incubator-trafodion/core/sqf/tmp Removing old monitor.port* files from /home/trafdeveloper/mysource/incubator-trafodion/core/sqf/tmp Executing sqipcrm (output to sqipcrm.out) Starting the SQ Environment (Executing /home/trafdeveloper/mysource/incubator-trafodion/core/sqf/sql/scripts/gomon.cold) Background SQ Startup job (pid: 15755) . . . You can monitor the SQ shell log file : /home/trafdeveloper/mysource/incubator-trafodion/core/sqf/logs/sqmon.log Startup time 0 hour(s) 0 minute(s) 59 second(s) $
- Initialize Trafodion
The final step is to initialize the Trafodion database.
Code Block | ||||
---|---|---|---|---|
| ||||
$ sqlci Apache Trafodion Conversational Interface 1.3.0 Copyright (c) 2015 Apache Software Foundation >>initialize trafodion; --- SQL operation complete. >> exit; End of MXCI Session $ |
Next Steps
At this point, you should have a running Trafodion Test Environment. The following chapters provide additional guidance for how to use the Trafodion Test Environment:
Test: Describes the Trafodion test libraries.
Manage Test Environment: Describes how to manage the test environment including tasks such as starting/stopping Trafodion.
...