Versions Compared

Old Version 30

changes.mady.by.user Dima Ruban

Saved on

compared with

New Version 31

changes.mady.by.user Dima Ruban

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Version 0.4.1 - September 2017.

Introduction

We will be installing Metron 0.4.1 with HDP 2.5 on CentOS 7. We will also install MariaDB as a database for Metron REST. Additionally, we'll also install Apache NiFi.
I installed Metron in a test environment with 3 VMs to try it out as well as a single node. I'll try to write this guide so that the necessary steps can easily be adapted for other environments.

Environment

  • Single node: 4 8 CPUs, 16GB 32GB RAM.
  • Multiple nodes:
  • 3 4 VMs, 2 4 CPUs per VM and 8 16 GB RAM per VM.
  • Hosts:
  • 10.10.10.1 node1
  • 10.10.10.2 node2
  • 10.10.10.3 node3

...

  • 10.10.10.4 node4

Prerequisites

  • CentOS 7

  • Add the epel repository and update your system:

    Code Block
    yum install epel-release -y
yum update -y

...

  • Adjust limits to secure level ([link](https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_installing_manually_book/content/ref-729d1fb0-6d1b-459f-a18a-b5eba4540ab5.1.html)):

    Code Block
    ulimit -n 32768
ulimit -u 65536
echo -e "* - nofile 32768\n* - nproc 65536" >> /etc/security/limits.conf

  • Disable IPv6, leaving it enabled may force service to bind to IPv6 addresses only and thus resulting in inability to connect to it ([source link](https://wiki.centos.org/FAQ/CentOS6#head-d47139912868bcb9d754441ecb6a8a10d41781df)):

    Code Block
    sysctl -w net.ipv6.conf.all.disable_ipv6=1
sysctl -w net.ipv6.conf.default.disable_ipv6=1
echo -e "\n# Disable IPv6\nnet.ipv6.conf.all.disable_ipv6 = 1\nnet.ipv6.conf.default.disable_ipv6 = 1" >> /etc/sysctl.conf

  • Disable Transparent Hugepage. Add "transparent_hugepage=never" to the end of the kernel line in "/etc/default/grub" and reboot. (Ambari demands it, do we need to comply?)

    Code Block
    # Change the line:
GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=cl/root rd.lvm.lv=cl/swap rhgb quiet"
# To:
GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=cl/root rd.lvm.lv=cl/swap rhgb quiet transparent_hugepage=never"
# Afterwards, run:
grub2-mkconfig -o /boot/grub2/grub.cfg

    After reboot check that changes were applied (make sure that word "never" is selected in square-brackets):

    Code Block
    cat /sys/kernel/mm/transparent_hugepage/enabled
always madvise [never]

    Alternatively, if you do not want to mess with kernel parameters, you can create a new systemd service which disables this on each boot. Create the file "/etc/systemd/system/disable-thp.service" with the following content:

    Code Block
    [Unit]
Description=Disable Transparent Huge Pages (THP)

[Service]
Type=simple
ExecStart=/bin/sh -c "echo 'never' > /sys/kernel/mm/transparent_hugepage/enabled && echo 'never' > /sys/kernel/mm/transparent_hugepage/defrag"

[Install]
WantedBy=multi-user.target

    Restart systemd, start the new service and make sure the new service runs at startup:

    Code Block
    # systemctl daemon-reload

...

  • 
# systemctl start disable-thp

...

  • 
# systemctl enable disable-thp

Metron install pre-preparation

...

  • On all nodes Install pre-requisites for Ambari:

    Code Block
    yum install git wget curl rpm tar unzip scp bzip2 wget createrepo yum-utils ntp python-pip psutils python-psutil ntp libffi-devel gcc openssl-devel -y
pip install --upgrade pip
pip install requests

...