...

Panel

Tabs Container

id	build-tabs
direction	horizontal

Tabs Page

id	macos
title	MacOS (10.10+)

Install dependencies on MAC (with xcode installed)

Code Block

language	bash

brew install protobuf protobuf-c Gsasl openssl boost thrift json-c ccache snappy libyaml libevent python
brew tap brona/iproute2mac
brew install iproute2mac
brew install postgresql
brew install cmake
sudo pip install pygresql==4.0  (make sure specified version 4.0)
sudo pip install unittest2 pycrypto lockfile paramiko psi pyyaml
sudo pip install http://sourceforge.net/projects/pychecker/files/pychecker/0.8.19/pychecker-0.8.19.tar.gz/download
sudo pip install http://darcs.idyll.org/~t/projects/figleaf-0.6.1.tar.gz
brew uninstall postgresql

OS requirement

Use a text editor to edit the /etc/sysctl.conf file. Add or edit each of the following parameter definitions to set the required value.

kern.sysv.shmmax=2147483648

kern.sysv.shmmin=1

kern.sysv.shmmni=64

kern.sysv.shmseg=16

kern.sysv.shmall=524288

kern.maxfiles=65535

kern.maxfilesperproc=65536

Reboot to apply the change.

Install Xcode

Xcode includes the tools, compiler and SDK for building HAWQ.

Tabs Page

id	tab-yum
title	Redhat/Centos 7.X

Install dependencies on CentOS 7.X

Dependencies

(CentOS7 user can follow easy steps provided by Zhanwei Wang)

Code Block

language	bash

curl -L "https://bintray.com/wangzw/rpm/rpm" -o /etc/yum.repos.d/bintray-wangzw-rpm.repo
yum install -y epel-release
yum makecache
yum install -y man passwd sudo tar which git mlocate links make bzip2 net-tools \
  autoconf automake libtool m4 gcc gcc-c++ gdb bison flex cmake gperf maven indent \
  libuuid-devel krb5-devel libgsasl-devel expat-devel libxml2-devel \
  perl-ExtUtils-Embed pam-devel python-devel libcurl-devel snappy-devel \
  thrift-devel libyaml-devel libevent-devel bzip2-devel openssl-devel \
  openldap-devel protobuf-devel readline-devel net-snmp-devel apr-devel \
  libesmtp-devel xerces-c-devel python-pip json-c-devel libhdfs3-devel \
  apache-ivy java-1.7.0-openjdk-devel \
  openssh-clients openssh-server
yum install -y postgresql-devel
pip --retries=50 --timeout=300 install pg8000 simplejson unittest2 pycrypto pygresql pyyaml lockfile paramiko psi
pip --retries=50 --timeout=300 install http://darcs.idyll.org/~t/projects/figleaf-0.6.1.tar.gz
pip --retries=50 --timeout=300 install http://sourceforge.net/projects/pychecker/files/pychecker/0.8.19/pychecker-0.8.19.tar.gz/download
yum erase -y postgresql postgresql-libs postgresql-devel

OS requirement

use a text editor to edit the /etc/sysctl.conf file. Add or edit each of the following parameter definitions to set the required value.

     kernel.shmmax = 1000000000
     kernel.shmmni = 4096
     kernel.shmall = 4000000000
     kernel.sem = 250 512000 100 2048
     kernel.sysrq = 1
     kernel.core_uses_pid = 1
     kernel.msgmnb = 65536
     kernel.msgmax = 65536
     kernel.msgmni = 2048
     net.ipv4.tcp_syncookies = 0
     net.ipv4.ip_forward = 0
     net.ipv4.conf.default.accept_source_route = 0
     net.ipv4.tcp_tw_recycle = 1
     net.ipv4.tcp_max_syn_backlog = 200000
     net.ipv4.conf.all.arp_filter = 1
     net.ipv4.ip_local_port_range = 1281 65535
     net.core.netdev_max_backlog = 200000
     vm.overcommit_memory = 2
     fs.nr_open = 3000000
     kernel.threads-max = 798720
     kernel.pid_max = 798720
     # increase network
     net.core.rmem_max=2097152
     net.core.wmem_max=2097152

Execute the following command to apply your updated /etc/sysctl.conf file to the operating system configuration:
```
sysctl -p
```

Use a text editor to edit the /etc/security/limits.conf file. Add the following definitions in the exact order that they are listed

     * soft nofile 2900000
     * hard nofile 2900000
     * soft nproc 131072
     * hard nproc 131072

Tabs Page

title	Redhat/Centos 6.X
class	tabs-compile-dependencies

Build dependencies yourself ( tested on Redhat 6.X).

Dependencies

There are several dependencies (see the following table) you must install before building HAWQ. To build Apache HAWQ, gcc and some dependencies are needed. The libraries are tested on the given versions. Most of the dependencies can be installed through yum. Other dependencies should be installed through the source tarball. Typically you can use "./configure && make && make install" to install from source tarball.

Libraries that must be installed using source tarball.

Name	Version	Download URL
`boost`	`1.56`	http://sourceforge.net/projects/boost/files/boost/1.56.0/boost_1_56_0.tar.bz2
`thrift`	`0.9.1-1`	http://archive.apache.org/dist/thrift/0.9.1/thrift-0.9.1.tar.gz
`protobuf`	`2.5.0`	https://github.com/google/protobuf/tree/v2.5.0
`curl`	`7.44.0`	http://www.curl.haxx.se/download/curl-7.44.0.tar.gz
`libhdfs3`	`latest version`	https://github.com/PivotalRD/libhdfs3.git

You might need to run "ldconfig -p <LIBRARY_INSTALL_PATH>" after installing them.

For thrift build, you might need "--without-tests" for configure.

Libraries that can be installed through yum.

Name	Versoin
epel-release	6-8
make	`3.81`
gcc	`4.4.7`
gcc-c++	`4.4.7`
`gperf`	`3.0.4`
`snappy-devel`	`1.1.3`
`bzip2-devel`	`1.0.6`
`python-devel`	`2.6.2`
`libevent-devel`	`1.4.6`
`krb5-devel`	`1.11.3`
`libuuid-devel`	`2.26.2`
`libgsasl-devel`	`1.8.0`
`libxml2-devel`	`2.7.8`
`zlib-devel`	`1.2.3`
`readline-devel`	`6`
`openssl-devel`	`0.9.8`
`bison`
json-c-devel
`apr-devel`	`1.2.12`
`libyaml-devel`	`0.1.1`
`flex`

You will need to install python packages same as those which are required for Redhat/centos 7.

OS requirement

use a text editor to edit the /etc/sysctl.conf file. Add or edit each of the following parameter definitions to set the required value.

     kernel.shmmax = 1000000000
     kernel.shmmni = 4096
     kernel.shmall = 4000000000
     kernel.sem = 250 512000 100 2048
     kernel.sysrq = 1
     kernel.core_uses_pid = 1
     kernel.msgmnb = 65536
     kernel.msgmax = 65536
     kernel.msgmni = 2048
     net.ipv4.tcp_syncookies = 0
     net.ipv4.ip_forward = 0
     net.ipv4.conf.default.accept_source_route = 0
     net.ipv4.tcp_tw_recycle = 1
     net.ipv4.tcp_max_syn_backlog = 200000
     net.ipv4.conf.all.arp_filter = 1
     net.ipv4.ip_local_port_range = 1281 65535
     net.core.netdev_max_backlog = 200000
     vm.overcommit_memory = 2
     fs.nr_open = 3000000
     kernel.threads-max = 798720
     kernel.pid_max = 798720
     # increase network
     net.core.rmem_max=2097152
     net.core.wmem_max=2097152

Execute the following command to apply your updated /etc/sysctl.conf file to the operating system configuration:
```
sysctl -p
```

Use a text editor to edit the /etc/security/limits.conf file. Add the following definitions in the exact order that they are listed

     * soft nofile 2900000
     * hard nofile 2900000
     * soft nproc 131072
     * hard nproc 131072

Tabs Page

id	docker-tab
title	Docker (Redhat/Centos 7.X)

Build with Prebuilt Docker Image

Probably the simplest way to get started with the build is starting with the community developed docker image with all the project dependencies pre-installed.

To use the docker image follow the steps on: https://hub.docker.com/r/mayjojo/hawq-devel/

...

Code Block

language	bash

# The Apache HAWQ source code can be obtained from the the following link: 
# Apache Repo: https://git-wip-us.apache.org/repos/asf/incubator-hawq.git or 
# GitHub Mirror: https://github.com/apache/incubator-hawq. 
git clone https://git-wip-us.apache.org/repos/asf/incubator-hawq.git
 
# The code directory is incubator-hawq.
CODE_BASE=`pwd`/incubator-hawq
 
# Install Libyarn and Libhdfs3
cd $CODE_BASE/depends/libyarn
mkdir build
cd build
../bootstrap --prefix=/usr/local/
make -j8
make install
 
cd $CODE_BASE/depends/libhdfs3
mkdir build
cd build
../bootstrap --prefix=/usr/local/
make -j8
make install 

cd $CODE_BASE
 
# Run command to generate makefile.
./configure

# Or you 
 
# Run command to generate makefile.
./configure

# Or you could use --prefix=/hawq/install/path to change the Apache HAWQ install path, 
# and you can also add some optional components using options (--with-python --with-perl)
./configure --prefix=/hawq/install/path --with-python --with-perl

# You can also run the command with --help for more configuration.
./configure --help

#Note: If ./configure complains that libyarn is missing, it is provided under ./depends/libyarn. 
# Please follow the above steps to install libyarn. You may need to run "ldconfig -p <PATH>" after libyarn is installed.

# Run command to build and install
# To build concurrently , run make with -j option. For example, make -j8
# On Linux system without large memory, you will probably encounter error like
# "Error occurred during initialization of VM" "Could not reserve enough space for object heap";
# Try to set vm.overcommit_memory = 1 temporarily, and/or avoid "-j" build, and/or add more memory and then rebuild.
# On mac os, you will probably see this error: "'openssl/ssl.h' file not found". 
# "brew link openssl --force" should be able to solve the issue.
make -j8
 
# Install HAWQ
make install

...

Code Block

language	bash

psql -d postgres
create table t ( i int );
insert into t values(1);
insert into t select generate_series(1,10000);
select count(*) from t;

Test HAWQ

Code Block

language	bash

# Unit test. To do unit test, go to the src/backend and run unittest.
cd $CODE_BASE/src/backend
make unittest-check

# Installcheck-good test. After installing HAWQ, please ensure HDFS work before initializing HAWQ. 
source /install/dir/greenplum_path.sh
hawq init cluster
make installcheck-good

Space shortcuts

Page tree

Versions Compared

Old Version 109

New Version 110

Key

Install dependencies on MAC (with xcode installed)

OS requirement

Install Xcode

Install dependencies on CentOS 7.X

Dependencies

OS requirement

Build dependencies yourself ( tested on Redhat 6.X).

OS requirement

Build with Prebuilt Docker Image

Test HAWQ

Space shortcuts

Page tree

Page History

Versions Compared

Old Version 109

New Version 110

Key

Install dependencies on MAC (with xcode installed)

OS requirement

Install Xcode

Install dependencies on CentOS 7.X

Dependencies

OS requirement

Build dependencies yourself ( tested on Redhat 6.X).

OS requirement

Build with Prebuilt Docker Image

Test HAWQ