THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
1. Build HAWQ/PXF Ranger RPM package
- Step 1: Configuration operating system and install build dependency libraries.
Step 2: Prepare
Prepare aHAWQ source code tarball for rpm building.
a) Make sure the source code tarball name format is: apache
b) Make sure the extracted directory name format is: apacheNo Format git clone https://github.com/apache/hawq.git wget -O https://dist.apache.org/repos/dist/release/hawq/%{hawq_version}-incubating/apache-hawq-src-%{hawq_version}-incubating.tar.gz
c) Put the HAWQ source tarball to %{top_dir}mv apache-hawq-src-%{hawq_version}-incubating
folder. Or use "HAWQ_SOURCE_TARBALL_PATH" to specify the find path of HAWQ source tarball..tar.gz ./hawq/contrib/hawq-package
- Step 2: Configuration operating system and install dependency libraries.
- Step 3: Run 'make distclean' and './configure'.
- Step 4: Run 'make rpm' in the top directory of source code
- You can set environment variable "HAWQ_RELEASE_VERSION" to specify HAWQ version
/Step 3: Configure and generate rpm package.
No Format cd hawq make distclean ./configure make rpmStep 4: Check rpm packages tarball.
Step 5: You can find HAWQ/PFX/Ranger plugin rpms tarball at:%{top_dir}/No Format ls -lcontrib/hawq-package/apache-hawq-rpm-%{hawq_version}-incubating.tar.gz- Please refer to: https://github.com/apache/incubator-hawq/blob/master/contrib/hawq-package/README for details.
2. Install HAWQ/PXF/Ranger with RPM
Step 1: Configure operating system following instruction in
Appendix 1.- Step 2: Install runtime dependency libraries following instruction in Appendix 2.
Step 3: Install Apache Hadoop rpm packages using bigtop.
Download bigtop repo as /etc/yum.repos.d/bigtop.repo from https://www-us.apache.org/dyn/distcloser.lua/bigtop/bigtop-1.2.01/repos/centos7/
No Format sudo wget -O /etc/yum.repos.d/bigtop.repo httpshttp://www-us.apache.org/distftp.jaist.ac.jp/pub/apache/bigtop/bigtop-1.2.01/repos/centos7/bigtop.repo
Install Hadoop component including Hadoop, Hive, Hbase and Tomcat. Note that bigtop-tomcat version need to be >= 6.0.
No Format sudo yum install bigtop-tomcat hadoop\* hive\* hbase\*Install Rangerbigtop-tomcat if ranger plugin service is enabled. The version need to be 6.x.
No Format sudo yum install ranger bigtop-tomcat
Initialize Hadoop and start it.
No Format export JAVA_HOME= Java Home Directory export PATH=${JAVA_HOME}/bin:$PATH sudo /etc/init.d/hadoop-hdfs-namenode init sudo service hadoop-hdfs-namenode start sudo service hadoop-hdfs-datanode start sudo -u hdfs hdfs dfs -mkdir /hawq_default sudo -u hdfs hdfs dfs -chown gpadmin /hawq_defaultInstall Ranger 0.6 using source code following instructions in Apache Ranger 0.5.0 Installation
Step 4: Get Step 2: Get HAWQ/PXF/Ranger RPM package.
Download rpm binary release tarball from Apache HAWQ binary repository
No Format wget -O apache-hawq-srcrpm-%{hawq_version}-incubating.tar.gz https://dist.apache.org/repos/dist/release/incubator/hawq/%{hawq_version}-incubating/apache-hawq-srcrpm-%{hawq_version}-incubating.tar.gz
Or copy it if you compile it by yourself
No Format cp $HAWQ_SOURCE_DIR/contrib/hawq-package/apache-hawq-rpm-%{hawq_version}-incubating.tar.gz .
Step 35: Install HAWQ/PXF /Ranger runtime dependenciesRanger with RPM package.
Extract apache-hawq-rpm-%{hawq_version}-incubating.tar.gz and go to the directory ‘hawq_rpm_packages’
No Format tar -xzvf apache-hawq-rpm-%{hawq_version}-incubating.tar.gz; cd hawq_rpm_packagesInstall the rpm packages with below sequence
No Format sudo rpm -ivh apache-hawq-2.4.0.0-el7.x86_64.rpm sudo rpm -ivh hawq-ranger-plugin-2.4.0.0-1.el7.noarch.rpm sudo rpm -ivh apache-tomcat-7.0.62-el6.noarch.rpm sudo rpm -ivh pxf-service-3.3.0.0-1.el6.noarch.rpm sudo rpm -ivh pxf-hdfs-3.3.0.0-1.el6.noarch.rpm sudo rpm -ivh pxf-hive-3.3.0.0-1.el6.noarch.rpm sudo rpm -ivh pxf-hbase-3.3.0.0-1.el6.noarch.rpm sudo rpm -ivh pxf-jdbc-3.3.0.0-1.el6.noarch.rpm sudo rpm -ivh pxf-json-3.3.0.0-1.el6.noarch.rpm sudo rpm -ivh pxf-ignite-3.3.0.0-1.el6.noarch.rpm sudo rpm -ivh pxf-3.3.0.0-1.el6.noarch.rpmInicialize HAWQ cluster with gpadmin user
No Format su gpadmin source /usr/local/apache-hawq/greenplum_path.sh # Create password-less SSH connection between nodes in the cluster hawq ssh-exkeys -h localhost # Initialize HAWQ cluster hawq init clusterAnchor appendix1 appendix1
Appendix 1: Configure operating system.Configure system control parameters: edit /etc/sysctl.conf file to set parameters with values; then execute "sudo sysctl -p" to apply the changes.
No Format 1) Configuration operating system
No Format Edit /etc/sysctl.conf file to set parameters with values as below. Then execute "sudo sysctl -p" to apply the changes. kernel.shmmax = 1000000000 kernel.shmmni = 4096 kernel.shmall = 4000000000 kernel.sem = 250 512000 100 2048 kernel.sysrq = 1 kernel.core_uses_pid = 1 kernel.msgmnb = 65536 kernel.msgmax = 65536 kernel.msgmni = 2048 net.ipv4.tcp_syncookies = 0 net.ipv4.conf.default.accept_source_route = 0 net.ipv4.tcp_tw_recycle = 1 net.ipv4.tcp_max_syn_backlog = 200000 net.ipv4.conf.all.arp_filter = 1 net.ipv4.ip_local_port_range = 1281 65535 net.core.netdev_max_backlog = 200000 vm.overcommit_memory = 2 fs.nr_open = 3000000 kernel.threads-max = 798720 kernel.pid_max = 798720 # increase network net.core.rmem_max=2097152 net.core.wmem_max=2097152 Edit fs.file-max=3000000
Configure system limits: edit /etc/security/limits.conf
file
to
set belowparameters
in
extract
the
same order; then relogin to make them take effect.
StepNo Format order. Then relogin to make them take effect. * soft nofile 2900000 * hard nofile 2900000 * soft nproc 131072 * hard nproc 131072Anchor appendix2 appendix2
Appendix 2: Install
build dependencies of HAWQ, following instruction at:runtime dependency libraries.
wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm # For CentOs 7 the link isNo Format
yum makecache# Install epel For CentOS 7: sudo rpm -ivh https://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-8.noarch.rpm For RHEL 7: sudo rpm -ivh https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
On# For
redhat7RHEL
make sure enabled7,
# Otherwise yum will prompt some packages(e.g. gperf) not be foundenable rhel-7-server-extras-rpms and rhel-7-server-optional-rpms channel in /etc/yum.repos.d/redhat.repo
man passwd sudo tar which git mlocate links make bzip2 net-tools \ autoconf automake libtool m4 gcc gcc-c++ gdb bison flex gperf maven indent \ libuuid-devel krb5-develsudo yum install -y
-devellibgsasl
expat-devel libxml2-devel \ perl-ExtUtils-Embed pam
develthrift boost protobuf java-1.7.0-openjdk-devel python-
libcurlpip
develkrb5-
snappyserver
devel \ thrift-devel libyaml-devel libevent-devel bzip2-devel openssl-devel \ openldap-devel protobuf-devel readline-devel net-snmp-devel apr-devel \ libesmtp-devel python-pip json-c-devel \ java-1.7.0-openjdk-devel lcov cmake\ openssh-clients openssh-server perl-JSON perl-Env # need tomcat6 if enable-rps # download from http://archive.apache.org/dist/tomcat/tomcat-6/v6.0.44/krb5-
libs json-c net-snmp-libs python-devel gcc snappy # Install pycrypto sudo pip --retries=50 --timeout=300 install pycrypto
Step 4: Install HAWQ/PXF Ranger with RPM package
Extract apache-hawq-rpm-2.2.0.0-incubating.tar.gz and go to the directory ‘hawq_rpm_packages’
No Format tar -xzvf apache-hawq-rpm-2.2.0.0-incubating.tar.gz; cd hawq_rpm_packages
Install the rpm packages with below sequence
(Requirement: you need to have gcc and python-devel installed.) |