...
Contributed by Laurens Vets <laurens@daemon.be>.
Version 0.4.1 - September 2017.2 - January 2018
Introduction
We will be installing Metron 0.4.1 with HDP 2.5 on CentOS 7. We will also install MariaDB as a database for Metron REST. Additionally, we'll also install Apache NiFi.
I installed Metron in a test environment with 4 VMs to try it out as well as a single node. I'll try to write this guide so that the necessary steps can easily be adapted for other environments.
...
Adjust limits to secure level ([link](https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_installing_manually_book/content/ref-729d1fb0-6d1b-459f-a18a-b5eba4540ab5.1.html)):
Code Block ulimit -n 32768 ulimit -u 65536 echo -e "* - nofile 32768\n* - nproc 65536" >> /etc/security/limits.conf
Disable IPv6, leaving it enabled may force service to bind to IPv6 addresses only and thus resulting in inability to connect to it ([source link](https://wiki.centos.org/FAQ/CentOS6#head-d47139912868bcb9d754441ecb6a8a10d41781df)):
Code Block sysctl -w net.ipv6.conf.all.disable_ipv6=1 sysctl -w net.ipv6.conf.default.disable_ipv6=1 echo -e "\n# Disable IPv6\nnet.ipv6.conf.all.disable_ipv6 = 1\nnet.ipv6.conf.default.disable_ipv6 = 1" >> /etc/sysctl.conf
Disable Transparent Hugepage. Add "transparent_hugepage=never" to the end of the kernel line in "/etc/default/grub" and reboot. (Ambari demands it, do we need to comply?)
Code Block # Change the line: GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=cl/root rd.lvm.lv=cl/swap rhgb quiet" # To: GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=cl/root rd.lvm.lv=cl/swap rhgb quiet transparent_hugepage=never" # Afterwards, run: grub2-mkconfig -o /boot/grub2/grub.cfg
After reboot check that changes were applied (make sure that word "never" is selected in square-brackets):
Code Block cat /sys/kernel/mm/transparent_hugepage/enabled always madvise [never]
Alternatively, if you do not want to mess with kernel parameters, you can create a new systemd service which disables this on each boot. Create the file "/etc/systemd/system/disable-thp.service" with the following content:
Code Block [Unit] Description=Disable Transparent Huge Pages (THP) [Service] Type=simple ExecStart=/bin/sh -c "echo 'never' > /sys/kernel/mm/transparent_hugepage/enabled && echo 'never' > /sys/kernel/mm/transparent_hugepage/defrag" [Install] WantedBy=multi-user.target
Restart systemd, start the new service and make sure the new service runs at startup:
Code Block # systemctl daemon-reload # systemctl start disable-thp # systemctl enable disable-thp
Disable SELinux (is a must to install Ambari and build Metron:
Code Block setenforce 0 sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
...
Download and install Maven 3.3.9:
Code Block wget httphttps://archive.apache.volia.netorg/dist/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz tar -zxf apache-maven-3.3.9-bin.tar.gz mv apache-maven-3.3.9 /opt PATH=/opt/apache-maven-3.3.9/bin:$PATH echo 'export PATH=/opt/apache-maven-3.3.9/bin:$PATH' > /etc/profile.d/maven.sh chmod +x /etc/profile.d/maven.sh
...
Install the database we will use for Metron REST on Master node:
Code Block yum install mariadb-server -y
Install JAVA MySQL connector on all nodes:
Code Block yum install mysql-connector-java -y
Configure
...
database for Metron REST
...
If you haven't run `mysql_secure_installation` after the database installation, do that first:
Code Block |
---|
systemctl start mariadb systemctl enable mariadb systemctl status mariadb mysql_secure_installation |
Should produce following output:
Code Block |
---|
NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MySQL SERVERS IN PRODUCTION USE! PLEASE READ EACH STEP CAREFULLY! In order to log into MySQL to secure it, we'll need the current password for the root user. If you've just installed MySQL, and you haven't set the root password yet, the password will be blank, so you should just press enter here. Enter current password for root (enter for none): OK, successfully used password, moving on... Setting the root password ensures that nobody can log into the MySQL root user without the proper authorisation. Set root password? [Y/n] New password: Re-enter new password: Password updated successfully! Reloading privilege tables.. ... Success! By default, a MySQL installation has an anonymous user, allowing anyone to log into MySQL without having to have a user account created for them. This is intended only for testing, and to make the installation go a bit smoother. You should remove them before moving into a production environment. Remove anonymous users? [Y/n] n ... skipping. Normally, root should only be allowed to connect from 'localhost'. This ensures that someone cannot guess at the root password from the network. Disallow root login remotely? [Y/n] ... Success! By default, MySQL comes with a database named 'test' that anyone can access. This is also intended only for testing, and should be removed before moving into a production environment. Remove test database and access to it? [Y/n] - Dropping test database... ERROR 1008 (HY000) at line 1: Can't drop database 'test'; database doesn't exist ... Failed! Not critical, keep moving... - Removing privileges on test database... ... Success! Reloading the privilege tables will ensure that all changes made so far will take effect immediately. Reload privilege tables now? [Y/n] ... Success! All done! If you've completed all of the above steps, your MySQL installation should now be secure. Thanks for using MySQL! Cleaning up... |
...
Enable time sync, disable firewall and SElinux on every node (I know, but for the sake of simplicity, quickness & testing, I've disabled selinux):
Code Block systemctl enable ntpd systemctl start ntpd iptables -P INPUT ACCEPT iptables -P FORWARD ACCEPT iptables -P OUTPUT ACCEPT iptables -t nat -F iptables -t mangle -F iptables -F iptables -X iptables-save > /etc/sysconfig/iptables systemctl stop firewalld systemctl disable firewalld setenforce 0
Also, if you are using CentOS 7 and Python 2.7.5 and above you will encounter an error during ambari agent install in Ambari UI:
Code Block [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)
To fix it disable cert check in Python like this (reference link):
Code Block sed -i 's/verify=platform_default/verify=disable/' /etc/python/cert-verification.cfg
...