...
Contributed by Laurens Vets <laurens@daemon.be>.
Version 0.4.1 - September 2017.2 - January 2018
Introduction
We will be installing Metron 0.4.1 with HDP 2.5 on CentOS 7. We will also install MariaDB as a database for Metron REST. Additionally, we'll also install Apache NiFi.
I installed Metron in a test environment with 3 4 VMs to try it out as well as a single node. I'll try to write this guide so that the necessary steps can easily be adapted for other environments.
Environment
- Single node: 4 8 CPUs, 16GB 32GB RAM.
- Multiple nodes:
- 3 4 VMs, 2 4 CPUs per VM and 8 16 GB RAM per VM.
- Hosts:
10.10.10.1 node1
10.10.10.2 node2
10.10.10.3 node3
10.10.10.4 node4
Prerequisites
...
- CentOS 7
Add the epel repository and update your system:
Code Block yum install epel-release -y yum update -y
Set up passwordless SSH between our nodes. If passwordless ssh has not yet been set up within the cluster, then in main node generate key:
Code Block cat /dev/zero | ssh-keygen -q -N "" 2>/dev/null cat .ssh/id_rsa.pub >> .ssh/authorized_keys chmod 400 .ssh/authorized_keys
If you're not installing on a single node, add this newly generated key to all the slave nodes:
Code Block ssh-copy-id -i ~/.ssh/id_rsa.pub <replace_with_node_ip>
...
Adjust limits to secure level ([link](https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_installing_manually_book/content/ref-729d1fb0-6d1b-459f-a18a-b5eba4540ab5.1.html)):
Code Block ulimit -n 32768 ulimit -u 65536 echo -e "* - nofile 32768\n* - nproc 65536" >> /etc/security/limits.conf
Disable IPv6, leaving it enabled may force service to bind to IPv6 addresses only and thus resulting in inability to connect to it (
[](https://wiki.centos.org/FAQ/CentOS6#head-d47139912868bcb9d754441ecb6a8a10d41781df)
):
Code Block sysctl -w net.ipv6.conf.all.disable_ipv6=1 sysctl -w net.ipv6.conf.default.disable_ipv6=1 echo -e "\n# Disable IPv6\nnet.ipv6.conf.all.disable_ipv6 = 1\nnet.ipv6.conf.default.disable_ipv6 = 1" >> /etc/sysctl.conf
Disable Transparent Hugepage. Add "transparent_hugepage=never" to the end of the kernel line in "/etc/default/grub" and reboot. (Ambari demands it, do we need to comply?)
Code Block # Change the line: GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=cl/root rd.lvm.lv=cl/swap rhgb quiet" # To: GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=cl/root rd.lvm.lv=cl/swap rhgb quiet transparent_hugepage=never" # Afterwards, run: grub2-mkconfig -o /boot/grub2/grub.cfg
After reboot check that changes were applied (make sure that word "never" is selected in square-brackets):
Code Block cat /sys/kernel/mm/transparent_hugepage/enabled always madvise [never]
Alternatively, if you do not want to mess with kernel parameters, you can create a new systemd service which disables this on each boot. Create the file "/etc/systemd/system/disable-thp.service" with the following content:
Code Block [Unit] Description=Disable Transparent Huge Pages (THP) [Service] Type=simple ExecStart=/bin/sh -c "echo 'never' > /sys/kernel/mm/transparent_hugepage/enabled && echo 'never' > /sys/kernel/mm/transparent_hugepage/defrag" [Install] WantedBy=multi-user.target
Restart systemd, start the new service and make sure the new service runs at startup:
Code Block # systemctl daemon-reload
...
# systemctl start disable-thp
...
# systemctl enable disable-thp
Disable SELinux (is a must to install Ambari and build Metron:
Code Block setenforce 0 sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
Metron install pre-preparation
...
On all nodes Install pre-requisites for Ambari:
Code Block yum install git wget curl rpm tar unzip scp bzip2 wget createrepo yum-utils ntp python-pip psutils python-psutil ntp libffi-devel gcc openssl-devel -y pip install --upgrade pip pip install requests
...
Download and install Maven 3.3.9:
Code Block wget httphttps://archive.apache.volia.netorg/dist/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz tar -zxf apache-maven-3.3.9-bin.tar.gz mv apache-maven-3.3.9 /opt PATH=/opt/apache-maven-3.3.9/bin:$PATH echo 'export PATH=/opt/apache-maven-3.3.9/bin:$PATH' > /etc/profile.d/maven.sh chmod +x /etc/profile.d/maven.sh
Check whether Maven works:
Code Block source /etc/profile.d/maven.sh mvn -V
You should see something similar to:
...
Code Block # mvn -V
...
Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T08:41:47-08:00)
...
Maven home: /opt/apache-maven-3.3.9
...
Java version: 1.8.0_131, vendor: Oracle Corporation
...
Java home: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.131-3.b12.el7_3.x86_64/jre
...
Default locale: en_US, platform encoding: UTF-8
...
OS name: "linux", version: "3.10.0-514.16.1.el7.x86_64", arch: "amd64", family: "unix"
...
[INFO] Scanning for projects...
...
[INFO] ------------------------------------------------------------------------
...
[INFO] BUILD FAILURE
...
[INFO] ------------------------------------------------------------------------
...
[INFO] Total time: 0.083 s
...
[INFO] Finished at: 2017-06-06T09:59:03-07:00
...
[INFO] Final Memory: 13M/479M
...
[INFO] ------------------------------------------------------------------------
...
[ERROR] No goals have been specified for this build. You must specify a valid lifecycle phase or a goal in the format <plugin-prefix>:<goal> or <plugin-group-id>:<plugin-artifact-id>[:<plugin-version>]:<goal>. Available lifecycle phases are: validate, initialize, generate-sources, process-sources, generate-resources, process-resources, compile, process-classes, generate-test-sources, process-test-sources, generate-test-resources, process-test-resources, test-compile, process-test-classes, test, prepare-package, package, pre-integration-test, integration-test, post-integration-test, verify, install, deploy, pre-clean, clean, post-clean, pre-site, site, post-site, site-deploy. -> [Help 1]
...
[ERROR]
...
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
...
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
...
[ERROR]
...
[ERROR] For more information about the errors and possible solutions, please read the following articles:
...
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/NoGoalSpecifiedException
...
On Ambari On Ambari node install and enable docker (we will need it to build Metron mpack for Ambari):
Code Block yum install docker-io -y systemctl start docker
...
Install the database we will use for Metron REST on Master node:
Code Block yum install mariadb-server -y
Install JAVA MySQL connector on all nodes:
Code Block yum install mysql-connector-java -y
Configure
...
database for Metron REST
...
If you haven't run `mysql_secure_installation` after the database installation, do that first:
Code Block |
---|
systemctl start mariadb
systemctl enable mariadb
systemctl status mariadb
mysql_secure_installation
|
...
Should produce following output:
Code Block |
---|
NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MySQL |
...
SERVERS IN PRODUCTION USE! |
...
PLEASE READ EACH STEP CAREFULLY! |
...
In order to log into MySQL to secure it, we'll need the current |
...
password for the root user. |
...
If you've just installed MySQL, and |
...
you haven't set the root password yet, the password will be blank, |
...
so you should just press enter here. |
...
Enter current password for root (enter for none): |
...
OK, successfully used password, moving on... |
...
Setting the root password ensures that nobody can log into the MySQL |
...
root user without the proper authorisation. |
...
Set root password? [Y/n] |
...
New password: |
...
Re-enter new password: |
...
Password updated successfully! |
...
Reloading privilege tables.. |
...
... Success! |
...
By default, a MySQL installation has an anonymous user, allowing anyone |
...
to log into MySQL without having to have a user account created for |
...
them. |
...
This is intended only for testing, and to make the installation |
...
go a bit smoother. |
...
You should remove them before moving into a |
...
production environment. |
...
Remove anonymous users? [Y/n] n |
...
... skipping. |
...
Normally, root should only be allowed to connect from 'localhost'. |
...
This |
...
ensures that someone cannot guess at the root password from the network. |
...
Disallow root login remotely? [Y/n] |
...
... Success! |
...
By default, MySQL comes with a database named 'test' that anyone can |
...
access. |
...
This is also intended only for testing, and should be removed |
...
before moving into a production environment. |
...
Remove test database and access to it? [Y/n] |
...
- Dropping test database... |
...
ERROR 1008 (HY000) at line 1: Can't drop database 'test'; database doesn't exist |
...
... Failed! |
...
Not critical, keep moving... |
...
- Removing privileges on test database... |
...
... Success! |
...
Reloading the privilege tables will ensure that all changes made so far |
...
will take effect immediately. |
...
Reload privilege tables now? [Y/n] |
...
... Success! |
...
All done! |
...
If you've completed all of the above steps, your MySQL |
...
installation should now be secure. |
...
Thanks for using MySQL! |
...
Cleaning up... |
...
Build Metron code
Now we are going to start to building Metron . At the time of writing, Metron 0.4.1 was released.code. For advanced users: there seems to be a new way of building mpack and rpms
Clone Metron repo and switch to 0.4.1 release:
Code Block git clone https://github.com/apache/metron cd metron git checkout Metron_0.4.1
Build Metron with HDP 2.5 profile:
Code Block cd metron mvn clean package -DskipTests -T 2C -P HDP-2.5.0.0,mpack cd metron-deployment/packaging/docker/rpm-docker mvn clean install -DskipTests -PHDP-2.5.0.0
If for some reason, the rpm-docker fails with the message "/bin/bash: ./build.sh: Permission denied", try disabling selinux:
Code Block |
---|
setenforce 0 |
And run "mvn clean install -DskipTests -PHDP-2.5.0.0" mvn commands again.
On all nodes, create a localrepo directory and copy the RPMs from Ambari node there:
Code Block mkdir /localrepo cp -rp /root/metron/metron-deployment/packaging/docker/rpm-docker/RPMS/noarch/* /localrepo/ createrepo /localrepo
If you're doing a multi node install, also copy the packages to the other nodes:
Code Block ssh root@node2 mkdir /localrepo scp /localrepo/*rpm root@node2:/localrepo/ ssh root@node2 createrepo /localrepo
Make sure to do the above on each node.
Fetch & create logrotate script for Hadoop Services:
Code Block wget -O /etc/logrotate.d/metron-ambari https://raw.githubusercontent.com/apache/metron/master/metron-deployment/roles/ambari_common/templates/metron-hadoop-logrotate.yml sed -i 's/^ {{ hadoop_logrotate_frequency }}.*$/ daily/' /etc/logrotate.d/metron-ambari sed -i 's/^ rotate {{ hadoop_logrotate_retention }}.*$/ rotate 30/' /etc/logrotate.d/metron-ambari chmod 0644 /etc/logrotate.d/metron-ambari
Ambari 2.4 with HDP 2.5 install
Inspired by: [https://docs.hortonworks.com/HDPDocuments/Ambari-2.4.3.0/bk_ambari-installation/content/ch_Getting_Ready.html]
Enable time sync, disable firewall and SElinux on every node :(I know, but for the sake of simplicity, quickness & testing, I've disabled selinux):
Code Block Code Block systemctl enable ntpd systemctl start ntpd systemctl stop firewalld systemctl disable firewalld setenforce 0 (=> I know, but for the sake of simplicity, quickness & testing, I've disabled selinux.)iptables -P INPUT ACCEPT iptables -P FORWARD ACCEPT iptables -P OUTPUT ACCEPT iptables -t nat -F iptables -t mangle -F iptables -F iptables -X iptables-save > /etc/sysconfig/iptables systemctl stop firewalld systemctl disable firewalld setenforce 0
Also, if you are using CentOS 7 and Python 2.7.5 and above you will encounter an error during ambari agent install in Ambari UI:
Code Block [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)
To fix it disable cert check in Python like this (reference link):
Code Block sed -i 's/verify=platform_default/verify=disable/' /etc/python/cert-verification.cfg
Make sure each node can resolve every other node's hostname or add hostname of each node to `/etc/hosts` on every node. For example add following lines in /etc/hosts of each node:
Code Block 10.10.10.1 node1
...
10.10.10.2 node2
...
10.10.10.3 node3 10.10.10.4 node4
Where 10.10.10.1, 10.10.10.2, 10.10.10.3 and 10.10.10.
...
4 are the IP addresses of your nodes and "node1", "node2", "node3" and "
...
node4" are their respective hostnames.
On main node download and setup Ambari repo (you may replace the "2.4.23.0" with a newer Ambari version number):
Code Block wget -nv http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.4.3.0/ambari.repo -O /etc/yum.repos.d/ambari.repo
...
Add Metron service to Ambari by running mpack command (make sure to specify correct path to mpack in --mpack=):
Code Block ambari-server install-mpack --mpack=/root/metron/metron-deployment/packaging/ambari/metron-mpack/target/metron_mpack-0.4.01.0.tar.gz --verbose
Start Ambari:
Code Block ambari-server start
...
# cp /root/metron/metron-platform/metron-api/target/metron-api-0.4.01.jar /usr/metron/0.4.01/lib/
# wget -O /etc/init.d/pcapservice https://raw.githubusercontent.com/apache/metron/master/metron-deployment/roles/metron_pcapservice/templates/pcapservice
# sed -i 's/{{ pcapservice_jar_dst }}/\/usr\/metron\/0.4.01\/lib\/metron-api-0.4.01.jar/' /etc/init.d/pcapservice
# sed -i 's/{{ pcapservice_port }}/8081/' /etc/init.d/pcapservice
# sed -i 's/{{ query_hdfs_path }}/\/tmp/' /etc/init.d/pcapservice
# sed -i 's/{{ pcap_hdfs_path }}/\/apps\/metron\/pcap/' /etc/init.d/pcapservice
# chmod 755 /etc/init.d/pcapservice
# wget -O /etc/logrotate.d/metron-pcapservice https://raw.githubusercontent.com/apache/metron/master/metron-deployment/roles/metron_pcapservice/templates/metron-pcapservice-logrotate.yml
# sed -i 's/^ {{ metron_pcapservice_logrotate_frequency }}.*$/ daily/' /etc/logrotate.d/metron-pcapservice
# sed -i 's/^ rotate {{ metron_pcapservice_logrotate_retention }}.*$/ rotate 30/' /etc/logrotate.d/metron-pcapservice
# chmod 644 /etc/logrotate.d/metron-pcapservice
...
# yum install monit -y
# wget -O /etc/monitrc https://github.com/apache/metron/raw/master/metron-deployment/roles/monit/templates/monit/monit.conf
# sed -i 's/{{ inventory_hostname }}/<IP ADDRESS>/' /etc/monitrc
# sed -i 's/{{ monit_user }}/admin/' /etc/monitrc
# sed -i 's/{{ monit_pass }}/monit/' /etc/monitrc
# chmod 600 /etc/monitrc
# wget -O /etc/monit.d/pcap-replay.monit https://github.com/apache/metron/raw/master/metron-deployment/roles/monit/templates/monit/pcap-replay.monit
# chmod 644 /etc/monit.d/pcap-replay.monit
# wget -O /etc/monit.d/pcap-service.monit https://github.com/apache/metron/raw/master/metron-deployment/roles/monit/templates/monit/pcap-service.monit
# chmod 644 /etc/monit.d/pcap-service.monit
# wget -O /etc/monit.d/pycapa.monit https://github.com/apache/metron/raw/master/metron-deployment/roles/monit/templates/monit/pycapa.monit
# chmod 644 /etc/monit.d/pycapa.monit
# wget -O /etc/monit.d/snort.monit https://github.com/apache/metron/raw/master/metron-deployment/roles/monit/templates/monit/snort.monit
# chmod 644 /etc/monit.d/snort.monit
# wget -O /etc/monit.d/yaf.monit https://github.com/apache/metron/raw/master/metron-deployment/roles/monit/templates/monit/yaf.monit
# chmod 644 /etc/monit.d/yaf.monit
# wget -O /etc/monit.d/bro.monit https://github.com/apache/metron/raw/master/metron-deployment/roles/monit/templates/monit/bro.monit
# sed -i 's/^ with pidfile.*$/ with pidfile \/usr\/local\/bro\/spool\/bro\/\.pid/' /etc/monit.d/bro.monit
# chmod 644 /etc/monit.d/bro.monit
# systemctl enable monit
# systemctl start monit
# systemctl status monit
# monit reload
# monit stop all
# monit start all
# monit summary | tail -n +3 | awk -F"'" '{print $2}'
Miscellaneous Issues
...
- I had a problem with Zeppelin after rebooting this machine and had to manually create the Zeppelin run directory:
...