Page History

Written by Dima Kovalyov <dimdroll@gmail.com>.

Contributed by Laurens Vets <laurens@daemon.be>.

Version 0.34.4 - July 2017.2 - January 2018

Introduction

We will be installing Metron 0.4.0 1 with HDP 2.5 on CentOS 7. We will also install MariaDB as a database for Metron REST. Additionally, we'll also install Apache NiFi.
I installed Metron in a test environment with 3 4 VMs to try it out as well as a single node. I'll try to write this guide so that the necessary steps can easily be adapted for other environments.

Environment

Single node: 4 8 CPUs, 16GB 32GB RAM.

Multiple nodes:
3 4 VMs, 2 4 CPUs per VM and 8 16 GB RAM per VM.
Hosts:
10.10.10.1 node1
10.10.10.2 node2
10.10.10.3 node3
10.10.10.4 node4

Prerequisites

...

CentOS 7

Add the epel repository and update your system:
Code Block
yum install epel-release -y yum update -y

Set up passwordless SSH between our nodes. If passwordless ssh has not yet been set up within the cluster, then in main node generate key:
Code Block
cat /dev/zero | ssh-keygen -q -N "" 2>/dev/null cat .ssh/id_rsa.pub >> .ssh/authorized_keys chmod 400 .ssh/authorized_keys
If you're not installing on a single node, add this newly generated key to all the slave nodes:
Code Block
ssh-copy-id -i ~/.ssh/id_rsa.pub <replace_with_node_ip>

...

Adjust limits to secure level ([link](https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_installing_manually_book/content/ref-729d1fb0-6d1b-459f-a18a-b5eba4540ab5.1.html)):
Code Block
ulimit -n 32768 ulimit -u 65536 echo -e "* - nofile 32768\n* - nproc 65536" >> /etc/security/limits.conf

Disable IPv6, leaving it enabled may force service to bind to IPv6 addresses only and thus resulting in inability to connect to it (

[

source link

](https://wiki.centos.org/FAQ/CentOS6#head-d47139912868bcb9d754441ecb6a8a10d41781df)):

):

Code Block

sysctl -w net.ipv6.conf.all.disable_ipv6=1
sysctl -w net.ipv6.conf.default.disable_ipv6=1
echo -e "\n# Disable IPv6\nnet.ipv6.conf.all.disable_ipv6 = 1\nnet.ipv6.conf.default.disable_ipv6 = 1" >> /etc/sysctl.conf

Disable Transparent Hugepage. Add "transparent_hugepage=never" to the end of the kernel line in "/etc/default/grub" and reboot. (Ambari demands it, do we need to comply?)

Code Block

# Change the line:
GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=cl/root rd.lvm.lv=cl/swap rhgb quiet"
# To:
GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=cl/root rd.lvm.lv=cl/swap rhgb quiet transparent_hugepage=never"
# Afterwards, run:
grub2-mkconfig -o /boot/grub2/grub.cfg

After reboot check that changes were applied (make sure that word "never" is selected in square-brackets):

Code Block
cat /sys/kernel/mm/transparent_hugepage/enabled always madvise [never]

Alternatively, if you do not want to mess with kernel parameters, you can create a new systemd service which disables this on each boot. Create the file "/etc/systemd/system/disable-thp.service" with the following content:

Code Block

[Unit]
Description=Disable Transparent Huge Pages (THP)

[Service]
Type=simple
ExecStart=/bin/sh -c "echo 'never' > /sys/kernel/mm/transparent_hugepage/enabled && echo 'never' > /sys/kernel/mm/transparent_hugepage/defrag"

[Install]
WantedBy=multi-user.target

Restart systemd, start the new service and make sure the new service runs at startup:

Code Block
# systemctl daemon-reload

...

# systemctl start disable-thp

...

# systemctl enable disable-thp

Disable SELinux (is a must to install Ambari and build Metron:

Code Block
setenforce 0 sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config

Metron install pre-preparation

...

On all nodes Install pre-requisites for Ambari:

Code Block
yum install git wget curl rpm tar unzip scp bzip2 wget createrepo yum-utils ntp python-pip psutils python-psutil ntp libffi-devel gcc openssl-devel -y pip install --upgrade pip pip install requests

...

Download and install Maven 3.3.9:

Code Block

wget httphttps://archive.apache.volia.netorg/dist/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz
tar -zxf apache-maven-3.3.9-bin.tar.gz
mv apache-maven-3.3.9 /opt
PATH=/opt/apache-maven-3.3.9/bin:$PATH
echo 'export PATH=/opt/apache-maven-3.3.9/bin:$PATH' > /etc/profile.d/maven.sh
chmod +x /etc/profile.d/maven.sh

Check whether Maven works:
Code Block
source /etc/profile.d/maven.sh mvn -V
You should see something similar to:
Code Block

...

# mvn -V

...


Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T08:41:47-08:00)

...

Maven home: /opt/apache-maven-3.3.9

...


Java version: 1.8.0_131, vendor: Oracle Corporation

...


Java home: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.131-3.b12.el7_3.x86_64/jre

...


Default locale: en_US, platform encoding: UTF-8

...


OS name: "linux", version: "3.10.0-514.16.1.el7.x86_64", arch: "amd64", family: "unix"

...

[INFO] Scanning for projects...

...


[INFO] ------------------------------------------------------------------------

...

[INFO] BUILD FAILURE

...


[INFO] ------------------------------------------------------------------------

...

[INFO] Total time: 0.083 s

...


[INFO] Finished at: 2017-06-06T09:59:03-07:00

...

[INFO] Final Memory: 13M/479M

...


[INFO] ------------------------------------------------------------------------

...


[ERROR] No goals have been specified for this build. You must specify a valid lifecycle phase or a goal in the format <plugin-prefix>:<goal> or <plugin-group-id>:<plugin-artifact-id>[:<plugin-version>]:<goal>. Available lifecycle phases are: validate, initialize, generate-sources, process-sources, generate-resources, process-resources, compile, process-classes, generate-test-sources, process-test-sources, generate-test-resources, process-test-resources, test-compile, process-test-classes, test, prepare-package, package, pre-integration-test, integration-test, post-integration-test, verify, install, deploy, pre-clean, clean, post-clean, pre-site, site, post-site, site-deploy. -> [Help 1]

...

[ERROR]

...


[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.

...


[ERROR] Re-run Maven using the -X switch to enable full debug logging.

...

[ERROR]

...


[ERROR] For more information about the errors and possible solutions, please read the following articles:

...


[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/NoGoalSpecifiedException

...

On Ambari node install and enable docker (we will need it to build Metron mpack for Ambari):
Code Block
yum install docker-io -y systemctl start docker

...

Install the database we will use for Metron REST on Master node:
Code Block
yum install mariadb-server -y
Install JAVA MySQL connector on all nodes:
Code Block
yum install mysql-connector-java -y

Configure

...

database for Metron REST

...

If you haven't run `mysql_secure_installation` after the database installation, do that first:

Code Block
systemctl start mariadb systemctl enable mariadb systemctl status mariadb mysql_secure_installation

...

Should produce following output:

Code Block
NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MySQL

...


      SERVERS IN PRODUCTION USE!

...

  PLEASE READ EACH STEP CAREFULLY!

...



In order to log into MySQL to secure it, we'll need the current

...


password for the root user.

...

  If you've just installed MySQL, and

...


you haven't set the root password yet, the password will be blank,

...


so you should just press enter here.

...



Enter current password for root (enter for none):

...


OK, successfully used password, moving on...

...



Setting the root password ensures that nobody can log into the MySQL

...


root user without the proper authorisation.

...



Set root password? [Y/n]

...


New password:

...


Re-enter new password:

...


Password updated successfully!

...


Reloading privilege tables..

...


 ... Success!

...




By default, a MySQL installation has an anonymous user, allowing anyone

...


to log into MySQL without having to have a user account created for

...


them.

...

  This is intended only for testing, and to make the installation

...


go a bit smoother.

...

  You should remove them before moving into a

...


production environment.

...



Remove anonymous users? [Y/n] n

...


 ... skipping.

...



Normally, root should only be allowed to connect from 'localhost'.

...

  This

...


ensures that someone cannot guess at the root password from the network.

...



Disallow root login remotely? [Y/n]

...


 ... Success!

...


By default, MySQL comes with a database named 'test' that anyone can

...


access.

...

  This is also intended only for testing, and should be removed

...


before moving into a production environment.

...



Remove test database and access to it? [Y/n]

...


 - Dropping test database...

...


ERROR 1008 (HY000) at line 1: Can't drop database 'test'; database doesn't exist

...


 ... Failed!

...

  Not critical, keep moving...

...


 - Removing privileges on test database...

...


 ... Success!

...



Reloading the privilege tables will ensure that all changes made so far

...


will take effect immediately.

...



Reload privilege tables now? [Y/n]

...


 ... Success!

...



All done!

...

  If you've completed all of the above steps, your MySQL

...


installation should now be secure.

...



Thanks for using MySQL!

...




Cleaning up...

...

Build Metron code

Now we are going to start to building Metron . At the time of writing, Metron 0.4.0 was in the final stages of being released.
code. For advanced users: there seems to be a new way of building mpack and rpms

Clone Metron repo and switch to 0.4.1 release:
Code Block
git clone
On the main node, clone the Metron repository:
Code Block
git clone https://github.com/apache/metron

If you want to make sure you're on the 0.4.0 release branch, do:

...

cd metron git checkout Metron_0.4.

...

1

Build Metron with HDP 2.5 profile:

Code Block
cd metron mvn clean package -DskipTests -T 2C -P HDP-2.5.0.0,mpack cd metron-deployment/packaging/docker/rpm-docker mvn clean install -DskipTests -PHDP-2.5.0.0

If for some reason, the rpm-docker fails with the message "/bin/bash: ./build.sh: Permission denied", try disabling selinux:

Code Block
setenforce 0

And run "mvn clean install -DskipTests -PHDP-2.5.0.0" mvn commands again.

On all nodes, create a localrepo directory and copy the RPMs from Ambari node there:

Code Block
mkdir /localrepo cp -rp /root/metron/metron-deployment/packaging/docker/rpm-docker/RPMS/noarch/* /localrepo/ createrepo /localrepo

If you're doing a multi node install, also copy the packages to the other nodes:
Code Block
ssh root@node2 mkdir /localrepo scp /localrepo/*rpm root@node2:/localrepo/ ssh root@node2 createrepo /localrepo
Make sure to do the above on each node.

Fetch & create logrotate script for Hadoop Services:

Code Block

wget -O /etc/logrotate.d/metron-ambari https://raw.githubusercontent.com/apache/metron/master/metron-deployment/roles/ambari_common/templates/metron-hadoop-logrotate.yml
sed -i 's/^  {{ hadoop_logrotate_frequency }}.*$/  daily/' /etc/logrotate.d/metron-ambari
sed -i 's/^  rotate {{ hadoop_logrotate_retention }}.*$/  rotate 30/' /etc/logrotate.d/metron-ambari
chmod 0644 /etc/logrotate.d/metron-ambari

Ambari 2.4 with HDP 2.5 install

Inspired by: [https://docs.hortonworks.com/HDPDocuments/Ambari-2.4.3.0/bk_ambari-installation/content/ch_Getting_Ready.html]

Enable time sync, disable firewall and SElinux on every node (I know, but for the sake of simplicity, quickness & testing, I've disabled selinux):

Code Block

systemctl enable ntpd
systemctl start ntpd
systemctl stop firewalld
systemctl disable firewalld
setenforce 0 (=> I know, but for the sake of simplicity, quickness & testing, I've disabled selinux.)iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEPT
iptables -t nat -F
iptables -t mangle -F
iptables -F
iptables -X
iptables-save > /etc/sysconfig/iptables
systemctl stop firewalld
systemctl disable firewalld
setenforce 0

Also, if you are using CentOS 7 and Python 2.7.5 and above you will encounter an error during ambari agent install in Ambari UI:

Code Block
[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)

To fix it disable cert check in Python like this (reference link):

Code Block
sed -i 's/verify=platform_default/verify=disable/' /etc/python/cert-verification.cfg

Make sure
Make sure
each node can resolve every other node's hostname or add hostname of each node to `/etc/hosts` on every node. For example add following lines in /etc/hosts of each node:
Code Block
10.10.10.1 node1

...

10.10.10.2 node2

...

10.10.10.3 node3 10.10.10.4 node4
Where 10.10.10.1, 10.10.10.2, 10.10.10.3 and 10.10.10.

...

4 are the IP addresses of your nodes and "node1", "node2", "node3" and "

...

node4" are their respective hostnames.

On main node download and setup Ambari repo (you may replace the "2.4.23.0" with a newer Ambari version number):

Code Block
wget -nv http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.4.3.0/ambari.repo -O /etc/yum.repos.d/ambari.repo

...

Add Metron service to Ambari by running mpack command (make sure to specify correct path to mpack in --mpack=):

Code Block
ambari-server install-mpack --mpack=/root/metron/metron-deployment/packaging/ambari/metron-mpack/target/metron_mpack-0.4.01.0.tar.gz --verbose

Start Ambari:
Code Block
ambari-server start

...

# cp /root/metron/metron-platform/metron-api/target/metron-api-0.4.01.jar /usr/metron/0.4.01/lib/
# wget -O /etc/init.d/pcapservice https://raw.githubusercontent.com/apache/metron/master/metron-deployment/roles/metron_pcapservice/templates/pcapservice
# sed -i 's/{{ pcapservice_jar_dst }}/\/usr\/metron\/0.4.01\/lib\/metron-api-0.4.01.jar/' /etc/init.d/pcapservice
# sed -i 's/{{ pcapservice_port }}/8081/' /etc/init.d/pcapservice
# sed -i 's/{{ query_hdfs_path }}/\/tmp/' /etc/init.d/pcapservice
# sed -i 's/{{ pcap_hdfs_path }}/\/apps\/metron\/pcap/' /etc/init.d/pcapservice
# chmod 755 /etc/init.d/pcapservice
# wget -O /etc/logrotate.d/metron-pcapservice https://raw.githubusercontent.com/apache/metron/master/metron-deployment/roles/metron_pcapservice/templates/metron-pcapservice-logrotate.yml
# sed -i 's/^  {{ metron_pcapservice_logrotate_frequency }}.*$/  daily/' /etc/logrotate.d/metron-pcapservice
# sed -i 's/^  rotate {{ metron_pcapservice_logrotate_retention }}.*$/  rotate 30/' /etc/logrotate.d/metron-pcapservice
# chmod 644 /etc/logrotate.d/metron-pcapservice

...

# yum install monit -y
# wget -O /etc/monitrc https://github.com/apache/metron/raw/master/metron-deployment/roles/monit/templates/monit/monit.conf
# sed -i 's/{{ inventory_hostname }}/<IP ADDRESS>/' /etc/monitrc
# sed -i 's/{{ monit_user }}/admin/' /etc/monitrc
# sed -i 's/{{ monit_pass }}/monit/' /etc/monitrc
# chmod 600 /etc/monitrc
# wget -O /etc/monit.d/pcap-replay.monit https://github.com/apache/metron/raw/master/metron-deployment/roles/monit/templates/monit/pcap-replay.monit
# chmod 644 /etc/monit.d/pcap-replay.monit
# wget -O /etc/monit.d/pcap-service.monit https://github.com/apache/metron/raw/master/metron-deployment/roles/monit/templates/monit/pcap-service.monit
# chmod 644 /etc/monit.d/pcap-service.monit
# wget -O /etc/monit.d/pycapa.monit https://github.com/apache/metron/raw/master/metron-deployment/roles/monit/templates/monit/pycapa.monit
# chmod 644 /etc/monit.d/pycapa.monit
# wget -O /etc/monit.d/snort.monit https://github.com/apache/metron/raw/master/metron-deployment/roles/monit/templates/monit/snort.monit
# chmod 644 /etc/monit.d/snort.monit
# wget -O /etc/monit.d/yaf.monit https://github.com/apache/metron/raw/master/metron-deployment/roles/monit/templates/monit/yaf.monit
# chmod 644 /etc/monit.d/yaf.monit
# wget -O /etc/monit.d/bro.monit https://github.com/apache/metron/raw/master/metron-deployment/roles/monit/templates/monit/bro.monit
# sed -i 's/^  with pidfile.*$/  with pidfile \/usr\/local\/bro\/spool\/bro\/\.pid/' /etc/monit.d/bro.monit
# chmod 644 /etc/monit.d/bro.monit
# systemctl enable monit
# systemctl start monit
# systemctl status monit
# monit reload
# monit stop all
# monit start all
# monit summary | tail -n +3 | awk -F"'" '{print $2}'

Miscellaneous Issues

...

I had a problem with Zeppelin after rebooting this machine and had to manually create the Zeppelin run directory:
...
This was fixed by deleting the Kibana index ".kibana": "curl -XDELETE http://localhost:9200/.kibana"

Miscellaneous Services

Load the correct Elasticsearch template with:
"# curl -s -w "%{http_code}" -u <USERNAME>:<PASSWORD> -H "X-Requested-By: ambari" -X POST -d '{ "RequestInfo": { "context": "Install ES Template from REST", "command": "ELASTICSEARCH_TEMPLATE_INSTALL"},"Requests/resource_filters": [{"service_name": "METRON","component_name": "METRON_INDEXING","hosts" : "<HOSTNAME>"}]}' http://<AMBARI HOST>:8080/api/v1/clusters/<CLUSTERNAME>/requests"

...

# nifi-1.2.0/bin/nifi.sh install
# nifi-1.2.0/bin/nifi.sh start

Exposed Interfaces

In the end, you'll end up with a bunch of exposed UIs:
- Ambari: http://node1:8080/
- Kibana: http://node1:5000/
- Sensor Status (monit): http://node1:2812
- Elasticsearch: http://node1:9200/
- Storm UI: http://node1:8744/
- Metron REST interface: http://node1:8082/swagger-ui.html#/
- Management UI: http://node1:4200/ (user/password)
- Apache Nifi: http://node1:8089/nifi/
- Zookeeper: http://node1:2181
- Kafka: http://node1:6667

Space shortcuts

Page tree

Versions Compared

Old Version 29

New Version Current

Key

Introduction

Environment

Prerequisites

Metron install pre-preparation

Configure

database for Metron REST

Build Metron code

Ambari 2.4 with HDP 2.5 install

Miscellaneous Issues

I had a problem with Zeppelin after rebooting this machine and had to manually create the Zeppelin run directory:
...
This was fixed by deleting the Kibana index ".kibana": "curl -XDELETE http://localhost:9200/.kibana"

Miscellaneous Services

Exposed Interfaces

TROUBLESHOOTING

Space shortcuts

Page tree

Page History

Versions Compared

Old Version 29

New Version Current

Key

Introduction

Environment

Prerequisites

Metron install pre-preparation

Configure

database for Metron REST

Build Metron code

Ambari 2.4 with HDP 2.5 install

Miscellaneous Issues

I had a problem with Zeppelin after rebooting this machine and had to manually create the Zeppelin run directory:...This was fixed by deleting the Kibana index ".kibana": "curl -XDELETE http://localhost:9200/.kibana"

Miscellaneous Services

Exposed Interfaces

TROUBLESHOOTING

I had a problem with Zeppelin after rebooting this machine and had to manually create the Zeppelin run directory:
...
This was fixed by deleting the Kibana index ".kibana": "curl -XDELETE http://localhost:9200/.kibana"