Versions Compared

Old Version 26

changes.mady.by.user Ruilong Huo

Saved on

compared with

New Version 27

changes.mady.by.user Ruilong Huo

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  • Step 1: Install Apache Hadoop rpm packages using bigtop

    • Download bigtop repo as /etc/yum.repos.d/bigtop.repo from https://www-us.apache.org/dist/bigtop/bigtop-1.2.0/repos/centos7/

      No Format
      sudo wget -O /etc/yum.repos.d/bigtop.repo https://www-us.apache.org/dist/bigtop/bigtop-1.2.0/repos/centos7/bigtop.repo

    • Install Hadoop component including Hadoop, Hive, Hbase and Tomcat. Note that bigtop-tomcat version need to be >= 6.0.

      No Format
      sudo yum install bigtop-tomcat hadoop\* hive\* hbase\*

    • Install Ranger.

      No Format
      sudo yum install ranger

  • Step 2: Get HAWQ/PXF/Ranger RPM package

    • Download rpm binary release tarball from Apache HAWQ binary repository

      No Format
      wget -O apache-hawq-src-%{hawq_version}-incubating.tar.gz https://dist.apache.org/repos/dist/release/incubator/hawq/%{hawq_version}-incubating/apache-hawq-src-%{hawq_version}-incubating.tar.gz

    • Or copy it if you compile it by yourself

      No Format
      cp $HAWQ_SOURCE_DIR/contrib/hawq-package/apache-hawq-rpm-%{hawq_version}-incubating.tar.gz .

  • Step 3: Install HAWQ/PXF/Ranger runtime dependencies

     

    • Step 2: 1) Configuration operating system:

      No Format
      use a text editor to edit the Edit /etc/sysctl.conf file. Add or edit each of the following parameter definitions to set the required value. 


      to set parameters with values as below. Then execute "sudo sysctl -p" to apply the changes.

kernel.shmmax = 1000000000
     kernel.shmmni = 4096
     kernel.shmall = 4000000000
     kernel.sem = 250 512000 100 2048
     kernel.sysrq = 1
     kernel.core_uses_pid = 1
     kernel.msgmnb = 65536
     kernel.msgmax = 65536
     kernel.msgmni = 2048
     net.ipv4.tcp_syncookies = 0
     net.ipv4.conf.default.accept_source_route = 0
     net.ipv4.tcp_tw_recycle = 1
     net.ipv4.tcp_max_syn_backlog = 200000
     net.ipv4.conf.all.arp_filter = 1
     net.ipv4.ip_local_port_range = 1281 65535
     net.core.netdev_max_backlog = 200000
     vm.overcommit_memory = 2
     fs.nr_open = 3000000
     kernel.threads-max = 798720
     kernel.pid_max = 798720
     # increase network
     net.core.rmem_max=2097152
     net.core.wmem_max=2097152
Execute the following command to apply your updated /etc/sysctl.conf file to the operating system configuration:
sysctl -p


Use a text editor to edit the  
Edit /etc/security/limits.conf file. Addto theset followingbelow definitionsparameters in extract the exactsame order. thatThen theyrelogin areto listed

make them    take effect.

* soft nofile 2900000
     * hard nofile 2900000
     * soft nproc 131072
     * hard nproc 131072

    • Step 2: Install build dependencies of HAWQ, following instruction at:

      No Format
      wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
# For CentOs 7 the link is https://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-8.noarch.rpm
rpm -ivh epel-release-latest-7.noarch.rpm
yum makecache
# On redhat7, make sure enabled rhel-7-server-extras-rpms and rhel-7-server-optional-rpms channel in /etc/yum.repos.d/redhat.repo
# Otherwise yum will prompt some packages(e.g. gperf) not be found
yum install -y man passwd sudo tar which git mlocate links make bzip2 net-tools \
  autoconf automake libtool m4 gcc gcc-c++ gdb bison flex gperf maven indent \
  libuuid-devel krb5-devel libgsasl-devel expat-devel libxml2-devel \
  perl-ExtUtils-Embed pam-devel python-devel libcurl-devel snappy-devel \
  thrift-devel libyaml-devel libevent-devel bzip2-devel openssl-devel \
  openldap-devel protobuf-devel readline-devel net-snmp-devel apr-devel \
  libesmtp-devel python-pip json-c-devel \
  java-1.7.0-openjdk-devel lcov cmake\
  openssh-clients openssh-server perl-JSON perl-Env

# need tomcat6 if enable-rps
# download from http://archive.apache.org/dist/tomcat/tomcat-6/v6.0.44/

pip --retries=50 --timeout=300 install pycrypto


  • Step 4: Install HAWQ/PXF Ranger with RPM package

    • Extract apache-hawq-rpm-2.2.0.0-incubating.tar.gz and go to the directory ‘hawq_rpm_packages’

      No Format
      tar -xzvf apache-hawq-rpm-2.2.0.0-incubating.tar.gz; cd hawq_rpm_packages

    • Install the rpm packages with below sequence

      No Format
      rpm -ivh apache-hawq-2.2.0.0-el7.x86_64.rpm
rpm -ivh hawq-ranger-plugin-2.2.0.0-1.el7.centos.noarch.rpm
rpm -ivh apache-tomcat-7.0.62-el6.noarch.rpm
rpm -ivh pxf-service-3.2.1.0-1.el6.noarch.rpm
rpm -ivh pxf-hdfs-3.2.1.0-1.el6.noarch.rpm
rpm -ivh pxf-hive-3.2.1.0-1.el6.noarch.rpm
rpm -ivh pxf-hbase-3.2.1.0-1.el6.noarch.rpm
rpm -ivh pxf-jdbc-3.2.1.0-1.el6.noarch.rpm
rpm -ivh pxf-json-3.2.1.0-1.el6.noarch.rpm
rpm -ivh pxf-3.2.1.0-1.el6.noarch.rpm