Building the Software (Under construction ....)

This document describes the steps required to build ApacheTrafodion software.

Supported Platforms

Red Hat or Centos 6.x (6.4 or later) versions are supported as development and production platforms.

Required Software

Install the following packages via yum install <package>

alsa-lib-devel	ant	ant-nodeps
boost-devel	device-mapper-multipath	dhcp
gcc-c++	gd	glibc-devel.i686
graphviz-perl	gzip	java-1.7.0-openjdk-devel
openldap-devel	libaio-devel	libibcm.i686
libibumad-devel	libibumad-devel.i686	libiodbc
libiodbc-devel	librdmacm-devel	librdmacm-devel.i686
log4cxx	log4cxx-devel	lua-devel
lzo-minilzo	net-snmp-devel	net-snmp-perl
openldap-clients	openldap-devel.i686	openmotif
openssl-devel.i686	openssl-static	perl-Config-IniFiles
perl-DBD-SQLite	perl-Config-Tiny	perl-Expect
perl-IO-Tty	perl-Math-Calc-Units	perl-Params-Validate
perl-Parse-RecDescent	perl-TermReadKey	perl-Time-HiRes
protobuf-compiler	protobuf-devel	python-qpid
python-qpid-qmf	qpid-cpp-client	qpid-cpp-client-devel
qpid-cpp-client-ssl	qpid-cpp-server	qpid-cpp-server-ssl
qpid-qmf	qpid-tools	readline-devel
saslwrapper	sqlite-devel	flex
libXext-devel	libX11-devel	libXau-devel
unixODBC	unixODBC-devel	uuid-perl
xinetd	xerces-c-devel	libcurl-devel
libxml2-devel	cmake

Example yum install script:

    yum install alsa-lib-devel ant ant-nodeps boost-devel device-mapper-multipath dhcp gcc-c++ gd glibc-devel glibc-devel.i686 \
        graphviz-perl java-1.7.0-openjdk-devel libaio-devel libibcm.i686 libibumad-devel libibumad-devel.i686 \
        libiodbc libiodbc-devel librdmacm-devel librdmacm-devel.i686 log4cxx log4cxx-devel lua-devel lzo-minilzo net-snmp-devel \
        net-snmp-perl openldap-clients openldap-devel.i686 openmotif openssl-devel openssl-devel.i686 openssl-static \
        perl-Config-IniFiles perl-DBD-SQLite perl-Config-Tiny perl-Expect perl-IO-Tty perl-Math-Calc-Units \
        perl-Params-Validate perl-Parse-RecDescent perl-TermReadKey perl-Time-HiRes protobuf-compiler \
        protobuf-devel python-qpid python-qpid-qmf qpid-cpp-client qpid-cpp-client-devel qpid-cpp-client-ssl \
        qpid-cpp-server qpid-cpp-server-ssl qpid-qmf qpid-tools readline-devel saslwrapper sqlite-devel \
        unixODBC unixODBC-devel uuid-perl xinetd xerces-c-devel \
        openldap-devel flex libcurl-devel libxml2-devel cmake

Check the following before continuing

Java 1.7.x or greater must be installed. Ensure JAVA_HOME environment variable exists and set to your JDK installation.
Install openssl version "OpenSSL 1.0.1e-fips" or greater. To check your version do: "openssl version".
Install protocol buffers version "libprotoc 2.3.0" or greater. To check your version do: "protoc –version"
Install Apache Maven version 3.0.5 or greater and add it to your PATH. to check your version do: "mvn --version"
Download latest Apache Trafodion source from Apache Incubator https://github.com/apache/incubator-trafodion, un-tar the file into the <Trafodion download directory>
Download, build and install additional development tools via Additional Build Tools.

Note: Trafodion source needs to be downloaded before installing additional build tools if you are planning on running the Trafodion supplied convenience script to download needed packages.

<TBD, Java and openssl are downloaded through Yum. It looks like protocol buffers are part of Centos/Redhat, Do I need to add instructions on how to download and setup Maven?>

Build Steps

Set up Trafodion configuration file

<TDB: change hardcoded version number 1.2.0 to be configurable>

Additional development tools are required before building Trafodion as described Additional Build Tools. A convenience script script exists that downloads, installs, and builds all these tools in a common directory. If this convenience script is not used or if any of these additional build tools are not found in the expected location, then the Trafodion configuration file needs to be updated. The Trafodion configuration file template is located in <download directory>/apache-trafodion-1.2.0-incubating/core/sqf/LocalSettingsTemplate.sh. To change values, copy this file to your home directory and change its name to .trafodion. Edit the .trafodion file and update according to the instructions. Be sure to change the location of your TOOLSDIR to your <tools installation directory>.

   cp <traf download directory>/apache-trafodion-1.2.0-incubating/core/sqf/LocalSettingsTemplate.sh ~/.trafodion

For now, don't change locations for HADOOP_PREFIX, HBASE_HOME, and HIVE_HOME. There is a later step that describes this process.

Build Trafodion

Start a new ssh session

    cd <download directory>/apache-trafodion-1.2.0-incubating
    source ./env.sh

Build using one of the following options:

make all (Build Trafodion, DCS, REST) OR
make package (Build Trafodion, DCS, REST, Client drivers) OR
make package-all (Build Trafodion, DCS, REST, Client drivers and Tests for all components

Verify build by executing the following statement, this script should return over 90 components

    sqvers

Build using local Hadoop

This section describes the steps to use the Trafodion installation script called 'install_local_hadoop' that encapsulates building the product and starting the Hadoop eco-system. This script uses a Cloudera distribution.

Make sure you have set up password less authentication. You should be able to "ssh localhost" without having to enter a password

Install Hadoop eco-system

Start a new ssh session and set environment

    cd <download directory>/apache-trafodion-1.2.0-incubating
    source ./env.sh
    install_local_hadoop

Verify that build completed by running the following command - should report: 6 java servers and 2 mysqld processes are running

    swstatus

Note:

The 'install_local_hadoop' script downloads Hadoop, HBase, Hive, and MySql jar files from the internet. To avoid this overhead, you can download the required files into a separate directory and set the environment variable MY_LOCAL_SW_DIST to point to this directory.

The following options are available with 'install_local_hadoop'. Use the -p option if the default Hadoop ports are already in use on your machine:

‘install_local_hadoop' — will use default port numbers for all services

'install_local_hadoop -p fromDisplay' - will start Hadoop with a port number range determined from the DISPLAY environment variable OR

‘install_local_hadoop -p rand' — will start with any random port number range between 9000 and 49000 OR

‘install_local_hadoop -p < specify a port # >' — will start with port number specified OR

When this script completes, Hadoop, HBase, Hive, and MySql (used as Hive's metadata repository) have been installed and are started.

The 'install_local_hadoop' script also creates several helper scripts starting with "sw" in the $MY_SQROOT/sql/scripts directory including:

swhadoop swhbase swhdfs

swhive swjdbc swmysql

swmysqladmin swstartall swstarthadoop

swstarthbase swstartmysql swstatus

swstopall swstophadoop swstophbase

swstopmysql swuninstall_local_hadoop swurls.html

swyarn

To start/stop/check Hadoop environment using Trafodion supplied scripts, you can execute ‘swstartall’ , ‘swstopall’ and ‘swstatus’, and if you need to remove the installation, execute the 'swuninstall_local_hadoop'.

Do the build

Start a new ssh session and set environment

    cd <download directory>/apache-trafodion-1.2.0-incubating
    source ./env.sh

Build using one of the following options:

make all (Build Trafodion, DCS, REST) OR
make package (Build Trafodion, DCS, REST, Client drivers) OR
make package-all (Build Trafodion, DCS, REST, Client drivers and Tests for all components

Verify build by executing the following statement, this script should return over 90 components

    sqvers

Install Trafodion components

    install_traf_components

Test Trafodion build

If you built Trafodion using "Build using standard Apache installation", following the instructions in "Test using standard Apache installation". If you built Trafodion using the "Build using local Hadoop", follow the instructions in "Test using local Hadoop".

Test using standard Apache installation

Note: All tar files are created in ‘distribution’ folder located at the very top level (incubator-trafodion) during the make step.

To install a custom Trafodion component you can set various environment variable to overwrite the default tar files found in distribution folder.

Environment variables supported by install_traf_component script are :

DCS_TAR — Specify the fully qualified path of DCS tar file

REST_TAR — Specify the fully qualified path of REST tar file

PHX_TAR — Specify the fully qualified path of Phoenix test tar file

CLIENT_TAR — Specify the fully qualified path of Trafodion client tar file

DCSTEST_TAR — Specify the fully qualified path of DCS tests tar file

Follow installation instructions found on the Apache Trafodion wiki site - (Installation ) but modify the step that request you to download tar files from the download directory to use your generated the tar files.

Test using local Hadoop

This section describes how to start up Trafodion and run operations. It assumes that you have previously run the 'install_local_hadoop' script.

The 'install_local_hadoop' script configures everything needed to run Trafodion. It also starts Hadoop, Hive, MySql, and HBase.

Before starting, please check your ulimit. Some of them may need to be increased. If incorrect values are used, Trafodion may not start.

<TDB: should I add instructions on how to do this?>

The following are the recommended values.

ulimit –a

core file size (blocks, -c) 1000000
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 515196
max locked memory (kbytes, -l) 49595556
max memory size (kbytes, -m) unlimited
open files (-n) 32000
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 267263
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

Start Trafodion

Start a new ssh session and set environment, then perform a one-time step to set up run environment:

    cd <download directory>/apache-trafodion-1.2.0-incubating
    source ./env.sh
    cd $MY_SQROOT/sql/scripts
    sqgen

Important: After sqgen, exit all shells used for Trafodion, and source the env.sh file in a new shell.

Sometimes HBase does not get started correctly and needs to be restarted:

    swstatus

If HMaster is not started do:

    swstophbase
    swstarthbase

Finally, start the Trafodion processes:

    sqstart

Note: In case of any issues and if there is a need to stop and restart a specific Trafodion component, you can use the component based start/stop scripts.

Component	Start script	Stop script
For all of Trafodion	sqstart	sqstop
For DCS (Database Connectivity Service)	dcsstart	dcsstop
For REST server	reststart	reststop
For LOB server	lobstart	lobstop
For RMS server	rmsstart	rmsstop

Checking the status of Trafodion and its components

There are several health check scripts that are available which will provide the status of Trafodion. They are :

sqcheck (For all of Trafodion)
dcscheck (For Database Connectivity Service)
rmscheck (For RMS Server)

Creating Trafodion metadata

Trafodion is up and running, you can now start up a SQL command line interface and initialize Trafodion

Start a new ssh session and set environment and run the sqlci tool:

    cd <download directory>/apache-trafodion-1.2.0-incubating
    source ./env.sh
    sqlci

Perform the following statements:

    initialize trafodion;
    exit

Test your setup by using "sqlci" or "trafci" (uses DCS to connect to the SQL engine):

    get schemas;
    create table table1 (a int);
    invoke table1;
    insert into table1 values (1), (2), (3), (4);
    select * from table1;
    exit;

You are done and ready to go! <TBD: add reference to documentation>

Trafodion test packages

There are several helper scripts provided to run the tests for Trafodion components in your workspace. These scripts are generated based on the tar files that are made available during execution of install_traf_component script

swphoenix {t4 | t2 } — This script will run the Phoenix test using JDBC Type 4 driver or JDBC Type 2 driver

swjdbc — This script will run JDBC Type 4 tests

swpyodbc — This script will install Linux drivers run ODBC tests using Linux driver

Space shortcuts

Page tree