1. How to Build

Setup an environment with the dependencies installed

1.1 Get the code of HAWQ

Install dependencies on MAC (with xcode installed)

Panel

Tabs Container

id	build-tabs
direction	horizontal

Tabs Page

id	macos
title	MacOS (10.10+)

Code Block

language	bash

brew

install

The

protobuf

Apache

protobuf-c

HAWQ

Gsasl

source

openssl

code

boost

can

thrift

be

json-c

obtained

ccache

from

snappy

the

libyaml

the

libevent

following link: 
# Apache Repo: https://git.apache.org/repos/asf/hawq.git
# GitHub Mirror: https://github.com/apache/hawq.git
# Gitee Mirror: https://gitee.com/mirrors/hawq.git

git clone https://git.apache.org/repos/asf/hawq.git

1.2 Setup an environment with the dependencies installed

Panel

Tabs Container

id	build-tabs
direction	horizontal

Tabs Page

id	build-env-new
title	MacOS/Linux

Code Block

language	bash

git clone https://github.com/apache/hawq
cd hawq

# Download and setup pre-built dependency.
# This setup is required before each building process.
source .github/workflows/scripts/toolchain.sh

Tabs Page

id	macos
title	MacOS (10.10+)

Install dependencies on MAC (with xcode installed)

Make sure you have done: xcode-select --install to install developer tools

Code Block

language	bash

brew install Gsasl boost bison ccache snappy libyaml libevent cmake lcov apr apr-util iproute2mac glog lz4
brew install openssl protobuf protobuf-c thrift@0.9 libiconv

#install json-c
git clone https://github.com/json-c/json-c.git
cd json-c
git reset --hard json-c-0.12.1-20160607
sh autogen.sh && ./configure && make && sudo make install

brew install python
# make sure python version = 2.7

brew install perl
# make sure perl version < 5.30

curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
sudo python get-pip.py
sudo pip install cogapp
sudo pip install pycrypto

brew install jsoncpp
#To install jsoncpp, make sure its header files are installed in json/*.h.
#For macos, we use "brew install jsoncpp" to install jsoncpp, while its header files are in json/*.h as expected.
#For linux, we use "yum install jsoncpp" to install jsoncpp, its header files are in jsoncpp/json/*.h, so we need to 
#run "cp -rf jsoncpp/json ./" in jsonopp include path 

brew link --force thrift@0.9
ln -s /usr/local/opt/apr/libexec/lib/libapr-1.dylib /usr/local/lib/
ln -s /usr/local/opt/apr/libexec/bin/apr-1-config /usr/local/bin/apr-1-config
ln -s /usr/local/opt/openssl/lib/libcrypto.dylib /usr/local/lib/
ln -s /usr/local/opt/openssl/lib/libssl.dylib /usr/local/lib/
ln -s /usr/local/opt/openssl/bin/openssl /usr/local/bin/openssl
ln -s /usr/local/opt/libiconv/lib/libiconv.dylib /usr/local/lib/
ln -s /usr/local/opt/libiconv/lib/libcharset.dylib /usr/local/lib/
ln -s /usr/local/opt/libiconv/bin/iconv /usr/local/bin/iconv
cp /usr/local/opt/libiconv/include/* /usr/local/include/
export DEPENDENCY_INSTALL_PREFIX=/usr/local/opt/openssl
 
brew cask install java
# make sure jdk version is 7 or 8, otherwise must get java by the following website:
# http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html

brew install maven
# need tomcat6 if enable-rps
brew install tomcat@6

brew install cpanm
sudo cpanm JSON

# install protobuf and protobuf-c if brew install protobuf of protobuf-c failed
git clone https://github.com/protocolbuffers/protobuf.git &&
cd protobuf && git submodule update --init --recursive &&
./autogen.sh && ./configure && make &&
make check && make install && ldconfig

# install libesmtp manually
# get libesmtp from http://linuxfromscratch.org/blfs/view/svn/general/libesmtp.html
tar jxvf libesmtp-1.0.6.tar.bz2
cd libesmtp-1.0.6
./configure && make
sudo make install

Please refer to section below titled Running catalog tidycat perl modules for installing perl-JSON module on MAC/

Info

title	Note for Installing Dependencies

some dependencies require brew install <packagename> --universal, if you see HAWQ complains about an already installed package is required, try that.
El Capitan issues: boost cannot be installed as --universal (command shell will hang), you need follow manul steps http://www.boost.org/doc/libs/1_61_0/more/getting_started/unix-variants.html#easy-build-and-install
bison version on local MAC is too high. try bison --version to look at the version, if it is later than 3.0, try install bison with a lower version, such as 2.3. And rebuild HAWQ.
If you see openssl error, try:
brew link --force openssl
To make it work, here are two options:
1. Install the thirdparty library in system directory, i.e., /usr/local/bin,/usr/bin,/usr/local/lib,/usr/lib,/usr/local/include, etc.
2. Add executables and headers path to system path and export them accordingly.

OS requirement

Use a text editor to edit the /etc/sysctl.conf file. Add or edit each of the following parameter definitions to set the required value.

kern.sysv.shmmax=2147483648

kern.sysv.shmmin=1

kern.sysv.shmmni=64

kern.sysv.shmseg=16

kern.sysv.shmall=524288

kern.maxfiles=65535

kern.maxfilesperproc=65536

Reboot to apply the change.

Install Xcode and command line tools

After install/update xcode, please run ‘xcode-select --install’ to install command line tools, and then open xcode to make sure you have already installed it.

MUST: Turning Off Rootless System Integrity Protection in OS X El Capitan 10.11+

If not do this, you may encounter some tricky LIBRARY_PATH problems. e.g. HAWQ-513

Following below instructions: ( refer to http://osxdaily.com/2015/10/05/disable-rootless-system-integrity-protection-mac-os-x )

Reboot the Mac and hold down Command + R keys simultaneously after you hear the startup chime, this will boot OS X into Recovery Mode
When the “OS X Utilities” screen appears, pull down the ‘Utilities’ menu at the top of the screen instead, and choose “Terminal”
Type the following command into the terminal then hit return: csrutil disable; reboot

Tabs Page

id	tab-yum
title	Red Hat/CentOS 7.X

Install dependencies on Rad Hat/CentOS 7.X

Dependencies

Code Block

language	bash

wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
# For CentOs 7 the link is https://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-9.noarch.rpm
rpm -ivh epel-release-latest-7.noarch.rpm
yum makecache
# On redhat7, make sure enabled rhel-7-server-extras-rpms and rhel-7-server-optional-rpms channel in /etc/yum.repos.d/redhat.repo
# Otherwise yum will prompt some packages(e.g. gperf) not be found
yum install -y man passwd sudo tar which git mlocate links make bzip2 net-tools \
  autoconf automake libtool m4 gcc gcc-c++ gdb bison flex gperf maven indent \
  libuuid-devel krb5-devel libgsasl-devel expat-devel libxml2-devel \
  perl-ExtUtils-Embed pam-devel python-devel libcurl-devel snappy-devel \
  thrift-devel libyaml-devel libevent-devel bzip2-devel openssl-devel \
  openldap-devel protobuf-devel readline-devel net-snmp-devel apr-devel \
  libesmtp-devel python-pip json-c-devel \
  java-1.7.0-openjdk-devel lcov cmake3 \
  openssh-clients openssh-server perl-JSON perl-Env

# need tomcat6 if enable-rps
# download from http://archive.apache.org/dist/tomcat/tomcat-6/v6.0.44/

ln -s /usr/bin/cmake3 /usr/bin/cmake
pip --retries=50 --timeout=300 install pycrypto

OS requirement

use a text editor to edit the /etc/sysctl.conf file. Add or edit each of the following parameter definitions to set the required value.

     kernel.shmmax = 1000000000
     kernel.shmmni = 4096
     kernel.shmall = 4000000000
     kernel.sem = 250 512000 100 2048
     kernel.sysrq = 1
     kernel.core_uses_pid = 1
     kernel.msgmnb = 65536
     kernel.msgmax = 65536
     kernel.msgmni = 2048
     net.ipv4.tcp_syncookies = 0
     net.ipv4.conf.default.accept_source_route = 0
     net.ipv4.tcp_tw_recycle = 1
     net.ipv4.tcp_max_syn_backlog = 200000
     net.ipv4.conf.all.arp_filter = 1
     net.ipv4.ip_local_port_range = 1281 65535
     net.core.netdev_max_backlog = 200000
     vm.overcommit_memory = 2
     fs.nr_open = 3000000
     kernel.threads-max = 798720
     kernel.pid_max = 798720
     # increase network
     net.core.rmem_max=2097152

python brew tap brona/iproute2mac brew install iproute2mac brew install postgresql brew install cmake sudo pip install pygresql==4.0 (make sure specified version 4.0) sudo pip install unittest2 pycrypto lockfile paramiko psi pyyaml sudo pip install http://sourceforge.net/projects/pychecker/files/pychecker/0.8.19/pychecker-0.8.19.tar.gz/download sudo pip install http://darcs.idyll.org/~t/projects/figleaf-0.6.1.tar.gz brew uninstall postgresql

OS requirement

Use a text editor to edit the /etc/sysctl.conf file. Add or edit each of the following parameter definitions to set the required value.

kern.sysv.shmmax=2147483648

kern.sysv.shmmin=1

kern.sysv.shmmni=64

kern.sysv.shmseg=16

kern.sysv.shmall=524288

kern.maxfiles=65535

kern.maxfilesperproc=65536

Reboot to apply the change.

Install Xcode

Xcode includes the tools, compiler and SDK for building HAWQ.

Tabs Page

id	tab-yum
title	Redhat/Centos 7.X

Install dependencies on CentOS 7.X

Dependencies

(CentOS7 user can follow easy steps provided by Zhanwei Wang)

Code Block

language	bash

curl -L "https://bintray.com/wangzw/rpm/rpm" -o /etc/yum.repos.d/bintray-wangzw-rpm.repo
yum install -y epel-release
yum makecache
yum install -y man passwd sudo tar which git mlocate links make bzip2 net-tools \
  autoconf automake libtool m4 gcc gcc-c++ gdb bison flex cmake gperf maven indent \
  libuuid-devel krb5-devel libgsasl-devel expat-devel libxml2-devel \
  perl-ExtUtils-Embed pam-devel python-devel libcurl-devel snappy-devel \
  thrift-devel libyaml-devel libevent-devel bzip2-devel openssl-devel \
  openldap-devel protobuf-devel readline-devel net-snmp-devel apr-devel \
  libesmtp-devel xerces-c-devel python-pip json-c-devel libhdfs3-devel \
  apache-ivy java-1.7.0-openjdk-devel \
  openssh-clients openssh-server
yum install -y postgresql-devel
pip --retries=50 --timeout=300 install pg8000 simplejson unittest2 pycrypto pygresql pyyaml lockfile paramiko psi
pip --retries=50 --timeout=300 install http://darcs.idyll.org/~t/projects/figleaf-0.6.1.tar.gz
pip --retries=50 --timeout=300 install http://sourceforge.net/projects/pychecker/files/pychecker/0.8.19/pychecker-0.8.19.tar.gz/download
yum erase -y postgresql postgresql-libs postgresql-devel

OS requirement

use a text editor to edit the /etc/sysctl.conf file. Add or edit each of the following parameter definitions to set the required value.

     kernel.shmmax = 1000000000
     kernel.shmmni = 4096
     kernel.shmall = 4000000000
     kernel.sem = 250 512000 100 2048
     kernel.sysrq = 1
     kernel.core_uses_pid = 1
     kernel.msgmnb = 65536
     kernel.msgmax = 65536
     kernel.msgmni = 2048
     net.ipv4.tcp_syncookies = 0
     net.ipv4.ip_forward = 0
     net.ipv4.conf.default.accept_source_route = 0
     net.ipv4.tcp_tw_recycle = 1
     net.ipv4.tcp_max_syn_backlog = 200000
     net.ipv4.conf.all.arp_filter = 1
     net.ipv4.ip_local_port_range = 1281 65535
     net.core.netdev_max_backlog = 200000
     vm.overcommit_memory = 2
     fs.nr_open = 3000000
     kernel.threads-max = 798720
     kernel.pid_max = 798720
     # increase network
     net.core.rmem_max=2097152
     net.core.wmem_max=2097152

Execute the following command to apply your updated /etc/sysctl.conf file to the operating system configuration:
```
sysctl -p
```

Use a text editor to edit the /etc/security/limits.conf file. Add the following definitions in the exact order that they are listed

     * soft nofile 2900000
     * hard nofile 2900000
     * soft nproc 131072
     * hard nproc 131072

Tabs Page

title	Redhat/Centos 6.X
class	tabs-compile-dependencies

Build dependencies yourself ( tested on Redhat 6.X).

Dependencies

There are several dependencies (see the following table) you must install before building HAWQ. To build Apache HAWQ, gcc and some dependencies are needed. The libraries are tested on the given versions. Most of the dependencies can be installed through yum. Other dependencies should be installed through the source tarball. Typically you can use "./configure && make && make install" to install from source tarball.

Libraries that must be installed using source tarball.

Name	Version	Download URL
`json-c-devl`	0.9	http://oss.metaparadigm.com/json-c/json-c-0.9.tar.gz `(On centos 6.7 you can install it directly via yum.)`
`boost`	`1.56`	http://sourceforge.net/projects/boost/files/boost/1.56.0/boost_1_56_0.tar.bz2
`thrift`	`0.9.1-1`	http://archive.apache.org/dist/thrift/0.9.1/thrift-0.9.1.tar.gz (require boost 1.56)
`protobuf`	`2.5.0`	https://github.com/google/protobuf/tree/v2.5.0
`curl`	`7.44.0`	https://curl.haxx.se/download/curl-7.44.0.tar.gz
maven		http://repos.fedorapeople.org/repos/dchen/apache-maven/epel-apache-maven.repo

You might need to run "ldconfig -p <LIBRARY_INSTALL_PATH>" after installing them.

For thrift build, you might need "--without-tests" for configure.

Install maven:
sudo wget http://repos.fedorapeople.org/repos/dchen/apache-maven/epel-apache-maven.repo -O /etc/yum.repos.d/epel-apache-maven.repo
sudo sed -i s/\$releasever/6/g /etc/yum.repos.d/epel-apache-maven.repo
sudo yum install -y apache-maven

Install pip:

wget https://bootstrap.pypa.io/get-pip.py

python get-pip.py

Libraries that can be installed through yum.

Name	Versoin
epel-release	6-8
make	`3.81`
gcc	`4.4.7`
gcc-c++	`4.4.7`
`gperf`	`3.0.4`
`snappy-devel`	`1.1.3`
`bzip2-devel`	`1.0.6`
`python-devel`	`2.6.2`
`libevent-devel`	`1.4.6`
`krb5-devel`	`1.11.3`
`libuuid-devel`	`2.26.2`
`libgsasl-devel`	`1.8.0`
`libxml2-devel`	`2.7.8`
`zlib-devel`	`1.2.3`
`readline-devel`	`6`
`openssl-devel`	`0.9.8`
`bison`	1.875
`apr-devel`	`1.2.12`
`libyaml-devel`	`0.1.1`
`flex`	2.5.4

You will need to install python packages same as those which are required for Redhat/centos 7.

OS requirement

use a text editor to edit the /etc/sysctl.conf file. Add or edit each of the following parameter definitions to set the required value.

     kernel.shmmax = 1000000000
     kernel.shmmni = 4096
     kernel.shmall = 4000000000
     kernel.sem = 250 512000 100 2048
     kernel.sysrq = 1
     kernel.core_uses_pid = 1
     kernel.msgmnb = 65536
     kernel.msgmax = 65536
     kernel.msgmni = 2048
     net.ipv4.tcp_syncookies = 0
     net.ipv4.ip_forward = 0
     net.ipv4.conf.default.accept_source_route = 0
     net.ipv4.tcp_tw_recycle = 1
     net.ipv4.tcp_max_syn_backlog = 200000
     net.ipv4.conf.all.arp_filter = 1
     net.ipv4.ip_local_port_range = 1281 65535
     net.core.netdevwmem_max_backlog = 200000
     vm.overcommit_memory = 2
     =2097152

Execute the following command to apply your updated /etc/sysctl.conf file to the operating system configuration:
```
sysctl -p
```

Use a text editor to edit the /etc/security/limits.conf file. Add the following definitions in the exact order that they are listed
( Please make sure fs.nr_open

= 3000000 is applied before edit limits.conf, else you may not able to ssh to your instance.)

 3000000
     kernel.threads-max = 798720* soft nofile 2900000
     kernel.pid_max = 798720* hard nofile 2900000
     #* soft increasenproc network131072
     net.core.rmem_max=2097152
     net.core.wmem_max=2097152

Execute the following command to apply your updated /etc/sysctl.conf file to the operating system configuration:
```
sysctl -p
```

Use a text editor to edit the /etc/security/limits.conf file. Add the following definitions in the exact order that they are listed

     * soft nofile 2900000
     * hard nofile 2900000
     * soft nproc 131072
     * hard nproc 131072

Tabs Page

id	docker-tab
title	Docker (Redhat/Centos 7.X)

Build with Prebuilt Docker Image

Probably the simplest way to get started with the build is starting with the community developed docker image with all the project dependencies pre-installed.

To use the docker image follow the steps on: https://hub.docker.com/r/mayjojo/hawq-devel/

Install Hadoop

Please follow the steps here: https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SingleCluster.html

Note you might need to build hadoop from source on redhat/centos6.x if the downloaded hadoop package has higher

glibc version requirement. When that happens, you will probably see the warning below when running start-dfs.h.

" WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform"

You will also need to set the port for fs.defaultFS to 8020 in etc/hadoop/core-site.xml (The example above set it as 9000.)

HDFS is a must, but YARN is optional. YARN is only needed when you want to use YARN as the global resource manager.

Your need to verify your HDFS works.

Code Block

language	bash

# start HDFS
start-dfs.sh
 
# Do some basic tests to make sure HDFS works
hadoop fs -lr /
hadoop fs -mkdir /test
hadoop fs -put ./testfile /
hadoop fs -get /testfile .

Get the HAWQ code and Compile

Once you have an environment with the necessary dependencies installed and Hadoop is ready, the next step is to get the code and build HAWQ

Code Block

language	bash

# The Apache HAWQ source code can be obtained from the the following link: 
# Apache Repo: https://git-wip-us.apache.org/repos/asf/incubator-hawq.git or 
# GitHub Mirror: https://github.com/apache/incubator-hawq. 
git clone https://git-wip-us.apache.org/repos/asf/incubator-hawq.git
 
# The code directory is incubator-hawq.
CODE_BASE=`pwd`/incubator-hawq
 
cd $CODE_BASE
 
# Run command to generate makefile.
./configure

# Or you could use --prefix=/hawq/install/path to change the Apache HAWQ install path, 
# and you can also add some optional components using options (--with-python --with-perl)
./configure --prefix=/hawq/install/path --with-python --with-perl

# You can also run the command with --help for more configuration.
./configure --help


# Run command to build and install
# To build concurrently , run make with -j option. For example, make -j8
# On Linux system without large memory, you will probably encounter errors like
# "Error occurred during initialization of VM" and/or "Could not reserve enough space for object heap"
# and/or "out of memory", try to set vm.overcommit_memory = 1 temporarily, and/or avoid "-j" build,
# and/or add more memory and then rebuild.
# On mac os, you will probably see this error: "'openssl/ssl.h' file not found". 
# "brew link openssl --force" should be able to solve the issue.
make -j8
 
# Install HAWQ
make install

Init/Start/Stop HAWQ

Code Block

language	bash

# Before initializing HAWQ, you need to install HDFS and make sure it works.
# Besides you need to set password-less ssh on the systems.
source /hawq/install/path/greenplum_path.sh
hawq init cluster # after initialization, HAWQ is started by default

# Now you can stop/restart/start the cluster by using:  
hawq stop/restart/start cluster
 
# HAWQ master and segments are completely decoupled. So you can also init, start or stop the master and segments separately.
# For example, to init: hawq init master, then hawq init segment
# to stop: hawq stop master, then hawq stop segment
# to start: hawq start master, then hawq start segment

Connect and Run basic queries

Code Block

language	bash

psql -d postgres
create table t ( i int );
insert into t values(1);
insert into t select generate_series(1,10000);
select count(*) from t;

Test HAWQ

...

language	bash

...

* hard nproc 131072

Bison version: run bison --version to look at the version, if it is later than 3.0, try install bison with a lower version, such as 2.3. And rebuild HAWQ.

Tabs Page

title	Red Hat/CentOS 6.X
class	tabs-compile-dependencies

Build dependencies yourself ( tested on Red Hat 6.X).

Dependencies

There are several dependencies (see the following table) you must install before building HAWQ. To build Apache HAWQ, gcc and some dependencies are needed. The libraries are tested on the given versions. Most of the dependencies can be installed through yum. Other dependencies should be installed through the source tarball. Typically you can use "./configure && make && make install" to install from source tarball.

Libraries that must be installed using source tarball.

Name	Version	Download URL
`json-c-devel`	0.10	https://github.com/json-c/json-c/archive/json-c-0.10-20120530.tar.gz `(On centos 6.7 you may be able to install it directly via yum.)`
`boost`	`1.56`	http://sourceforge.net/projects/boost/files/boost/1.56.0/boost_1_56_0.tar.bz2
`thrift`	`0.9.1-1`	http://archive.apache.org/dist/thrift/0.9.1/thrift-0.9.1.tar.gz (require boost 1.56)
`protobuf`	`2.5.0`	https://github.com/google/protobuf/tree/v2.5.0
`curl`	`7.44.0`	https://curl.haxx.se/download/curl-7.44.0.tar.gz
maven		http://repos.fedorapeople.org/repos/dchen/apache-maven/epel-apache-maven.repo
cmake	>=3.0	https://cmake.org/files/

You might need to run "ldconfig -p <LIBRARY_INSTALL_PATH>" after installing them.

For thrift build, you might need "--without-tests" for configure.

Install maven:
sudo wget http://repos.fedorapeople.org/repos/dchen/apache-maven/epel-apache-maven.repo -O /etc/yum.repos.d/epel-apache-maven.repo
sudo sed -i s/\$releasever/6/g /etc/yum.repos.d/epel-apache-maven.repo
sudo yum install -y apache-maven

Install pip:

wget https://bootstrap.pypa.io/get-pip.py

python get-pip.py

pip --retries=50 --timeout=300 install pycrypto

Libraries that can be installed through yum.

Name	Version
epel-release	6-8
make	`3.81`
`gperf`	`3.0.4`
`snappy-devel`	`1.1.3`
`bzip2-devel`	`1.0.6`
`python-devel`	`2.6.2`
`libevent-devel`	`1.4.6`
`krb5-devel`	`1.11.3`
`libuuid-devel`	`2.26.2`
`libgsasl-devel`	`1.8.0`
`libxml2-devel`	`2.7.8`
`zlib-devel`	`1.2.3`
`readline-devel`	`6`
`openssl-devel`	`>=1.0.2`
`bison`	1.875
`apr-devel`	`1.2.12`
`libyaml-devel`	`0.1.1`
`flex`	>2.5.4
lcov	1.12
libesmtp-devel	1.0.4
perl-JSON	2.15
tomcat	6.0.44

Default version of gcc in Red Hat/CentOS 6.X is 4.4.7 or lower, you can quickly upgrade gcc following instructions below:

Code Block

language	bash

cd /etc/yum.repos.d
# make sure you have root permission
wget -O /etc/yum.repos.d/slc6-devtoolset.repo http://linuxsoft.cern.ch/cern/devtoolset/slc6-devtoolset.repo
# install higher version using devtoolset-2
yum install devtoolset-2-gcc devtoolset-2-binutils devtoolset-2-gcc-c++
# Start using software collections
scl enable devtoolset-2 bash

You will need to install python packages same as those which are required for Red Hat/CentOS 7.

OS requirement

use a text editor to edit the /etc/sysctl.conf file. Add or edit each of the following parameter definitions to set the required value.

     kernel.shmmax = 1000000000
     kernel.shmmni = 4096
     kernel.shmall = 4000000000
     kernel.sem = 250 512000 100 2048
     kernel.sysrq = 1
     kernel.core_uses_pid = 1
     kernel.msgmnb = 65536
     kernel.msgmax = 65536
     kernel.msgmni = 2048
     net.ipv4.tcp_syncookies = 0
     net.ipv4.conf.default.accept_source_route = 0
     net.ipv4.tcp_tw_recycle = 1
     net.ipv4.tcp_max_syn_backlog = 200000
     net.ipv4.conf.all.arp_filter = 1
     net.ipv4.ip_local_port_range = 1281 65535
     net.core.netdev_max_backlog = 200000
     vm.overcommit_memory = 2
     fs.nr_open = 3000000
     kernel.threads-max = 798720
     kernel.pid_max = 798720
     # increase network
     net.core.rmem_max=2097152
     net.core.wmem_max=2097152

Execute the following command to apply your updated /etc/sysctl.conf file to the operating system configuration:
```
sysctl -p
```

Use a text editor to edit the /etc/security/limits.conf file. Add the following definitions in the exact order that they are listed

     * soft nofile 2900000
     * hard nofile 2900000
     * soft nproc 131072
     * hard nproc 131072

Tabs Page

id	docker-tab
title	Docker (Red Hat/CentOS 7.X)

Build with Prebuilt Docker Image

Apache HAWQ source code contains the Dockerfiles to help developers to setup building and testing environment with docker.

To use the docker image follow the steps on: https://github.com/apache/hawq/tree/master/contrib/hawq-docker

1.3 Compile and Install HAWQ

Once you have an environment with the necessary dependencies installed and Hadoop is ready, the next step is to get the code and build HAWQ

Code Block

language	bash

# The code directory is hawq.
CODE_BASE=`pwd`/hawq
 
cd $CODE_BASE
 
# Run command to generate makefile.
./configure

# You can also run the command with --help for more configuration.
./configure --help

# Run command to build and install
# To build concurrently , run make with -j option. For example, make -j8
# On Linux system without large memory, you will probably encounter errors like
# "Error occurred during initialization of VM" and/or "Could not reserve enough space for object heap"
# and/or "out of memory", try to set vm.overcommit_memory = 1 temporarily, and/or avoid "-j" build,
# and/or add more memory and then rebuild.
# On mac os, you will probably see this error: "'openssl/ssl.h' file not found". 
# "brew link openssl --force" should be able to solve the issue.
make -j8
 
# Install HAWQ
make install

2. Init/Start/Stop HAWQ

2.1 Install and Start Hadoop

Please follow the steps here: https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SingleCluster.html

Note:

you might need to build hadoop from source on Red Hat/CentOS 6.x if the downloaded hadoop package has higher glibc version requirement. When that happens, you will probably see the warning below when running start-dfs.sh." WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform"
You will also need to set the port for fs.defaultFS to 8020 in etc/hadoop/core-site.xml (The example above set it as 9000.)
HDFS is a must, but YARN is optional. YARN is only needed when you want to use YARN as the global resource manager.
must setup passphraseless ssh, otherwise there will be some problems of "hawq init cluster" in the following step.

Your need to verify your HDFS works.

Code Block

language	bash

# start HDFS
start-dfs.sh
 
# Do some basic tests to make sure HDFS works
echo "test data" >> ./localfile
hadoop fs -mkdir /test
hadoop fs -put ./localfile /test
hadoop fs -ls /
hadoop fs -get /test/localfile ./hdfsfile

2.2 Init/Start/Stop HAWQ

Code Block

language	bash

# Before initializing HAWQ, you need to install HDFS and make sure it works.

source /hawq/install/path/greenplum_path.sh

# Besides you need to set password-less ssh on the systems.
# Exchange SSH keys between the hosts host1, host2, and host3:
hawq ssh-exkeys -h host1 -h host2 -h host3
hawq init cluster # after initialization, HAWQ is started by default

# Now you can stop/restart/start the cluster by using:  
hawq stop/restart/start cluster
 
# HAWQ master and segments are completely decoupled. So you can also init, start or stop the master and segments separately.
# For example, to init: hawq init master, then hawq init segment
# to stop: hawq stop master, then hawq stop segment
# to start: hawq start master, then hawq start segment

3. Connect and Run basic queries

Code Block

language	bash

psql -d postgres
create table t ( i int );
insert into t values(1);
insert into t select generate_series(1,10000);
select count(*) from t;

4. Query external hadoop data(optional)

You will need to use PXF to query external hadoop/hive/hbase data. Refer to PXF Build & Install document.

5. Test HAWQ

Code Block

language	bash

# Unit test. To do unit test, go to the src/backend and run unittest.
cd $CODE_BASE/src/backend
make unittest-check

# Code coverage
cd $CODE_BASE
./configure --enable-coverage --enable-debug (for debug build), or ./configure --enable-coverage (for opt build)
make -j8
make install
run some test to exercise hawq (i.e., unit test, install check, feature test, etc)
make coverage-show to see summary code coverage information in console, and detailed code coverage information in CodeCoverageReport (html format)
make coverage-show filter="./src/backend/executor/nodeAgg.c -d ./src/backend/commands" to see code coverage for specific files or directories
make coverage-reset to clear code coverage statistics
 
# Installcheck-good test. After installing HAWQ, please ensure HDFS work before initializing HAWQ. 
source /install/dir/greenplum_path.sh
hawq init cluster
make installcheck-good

# Feature test
cd $CODE_BASE
make feature-test
cd src/test/feature
./feature-test to run all feature test, or ./feature-test --gtest_filter=TestCommonLib.TestSqlUtil to run test suite TestCommonLib.TestSqlUtil

6. Running catalog tidycat perl modules(optional)

The JSON Perl Module is required to run the set of Perl scripts (src/include/catalog). The versioned JSON formatted catalog files are stored in tools/bin/gppylib/data/<version>.json. In order to install the JSON module, the developer will need to make the module available from CPAN. The following was validated on a Macbook Pro OS X 10.11.6 using the information from the Perl on Mac OSX section (http://www.cpan.org/modules/INSTALL.html). Below you will see the session which performs the following steps:

Validate JSON module is not in the environment. Receive appropriate error message.
Run cpan install JSON command to install the JSON Perl module. In the example below, the module is installed locally (local::lib) and not in the system's Perl installation.
Execute the environment variable updates added to the .bashrc file by the installation process.
Validate the tidycat.pl command can now be run without receiving error.

Note:

JSON Module version 2.27 and the latest 2.90 have been used to validate they generate the proper catalog JSON formatted file.
The scripts are essentially validating the evaluation of *require JSON* passes otherwise the error message is displayed:

Fatal Error: The required package JSON is not installed -- please download it from www.cpan.org

Code Block

00:02 $ perl tidycat.pl -dd 2.0.json -df json *.h
Fatal Error: The required package JSON is not installed -- please download it from www.cpan.org
00:02 $ 
00:02 $ cpan install JSON
	[many output stuff....]
00:05 $
00:05 $ perl tidycat.pl -dd foo.json -df json *.h
Fatal Error: The required package JSON is not installed -- please download it from www.cpan.org
00:05 $
00:05 $ PATH="/Users/espino/perl5/bin${PATH:+:${PATH}}"; export PATH;
00:05 $ PERL5LIB="/Users/espino/perl5/lib/perl5${PERL5LIB:+:${PERL5LIB}}"; export PERL5LIB;
00:05 $ PERL_LOCAL_LIB_ROOT="/Users/espino/perl5${PERL_LOCAL_LIB_ROOT:+:${PERL_LOCAL_LIB_ROOT}}"; export PERL_LOCAL_LIB_ROOT;
00:05 $ PERL_MB_OPT="--install_base \"/Users/espino/perl5\""; export PERL_MB_OPT;
00:05 $ PERL_MM_OPT="INSTALL_BASE=/Users/espino/perl5"; export PERL_MM_OPT;
00:05 $
00:05 $ perl tidycat.pl -dd 2.0.json -df json *.h
00:05 $

7. Build optional extension modules(optional)

Extension	How to enable	Pre-build steps on Mac
PL/R	./configure --with-r	#install R before build brew tap homebrew/science brew install r
PL/Python	./configure --with-python
PL/Java	./configure --with-java
PL/PERL	./configure --with-perl
pgcrypto	./configure --with-pgcrypto --with-openssl
gporca	./configure --enable-orca
rps	./configure --enable-rps	brew install tomcat@6

Space shortcuts

Page tree

Page History

Versions Compared

Old Version 114

New Version Current

Key

1. How to Build

Setup an environment with the dependencies installed

1.1 Get the code of HAWQ

Install dependencies on MAC (with xcode installed)

1.2 Setup an environment with the dependencies installed

Install dependencies on MAC (with xcode installed)

OS requirement

Install Xcode and command line tools

MUST: Turning Off Rootless System Integrity Protection in OS X El Capitan 10.11+

Install dependencies on Rad Hat/CentOS 7.X

Dependencies

OS requirement

OS requirement

Install Xcode

Install dependencies on CentOS 7.X

Dependencies

OS requirement

Build dependencies yourself ( tested on Redhat 6.X).

OS requirement

Build with Prebuilt Docker Image

Install Hadoop

Get the HAWQ code and Compile

Init/Start/Stop HAWQ

Connect and Run basic queries

Test HAWQ

Build dependencies yourself ( tested on Red Hat 6.X).

OS requirement

Build with Prebuilt Docker Image

1.3 Compile and Install HAWQ

2. Init/Start/Stop HAWQ

2.1 Install and Start Hadoop

2.2 Init/Start/Stop HAWQ

3. Connect and Run basic queries

4. Query external hadoop data(optional)

5. Test HAWQ

6. Running catalog tidycat perl modules(optional)

7. Build optional extension modules(optional)