Versions Compared

Old Version 29

changes.mady.by.user Dima Ruban

Saved on

compared with

New Version 30

changes.mady.by.user Dima Ruban

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Written by Dima Kovalyov <dimdroll@gmail.com>.

Contributed by Laurens Vets <laurens@daemon.be>.

Version 0.34.4 1 - July September 2017.

Introduction

We will be installing Metron 0.4.0 1 with HDP 2.5 on CentOS 7. We will also install MariaDB as a database for Metron REST. Additionally, we'll also install Apache NiFi.
I installed Metron in a test environment with 3 VMs to try it out as well as a single node. I'll try to write this guide so that the necessary steps can easily be adapted for other environments.

Environment

  • Single node: 4 CPUs, 16GB RAM.
  • Multiple nodes:
  • 3 VMs, 2 CPUs per VM and 8 GB RAM per VM.
  • Hosts:
  • 10.10.10.1 node1
  • 10.10.10.2 node2
  • 10.10.10.3 node3

Prerequisites:

  • CentOS 7

  • Add the epel repository and update your system:

    Code Block
    yum install epel-release -y
yum update -y

...

# systemctl daemon-reload
# systemctl start disable-thp
# systemctl enable disable-thp

Metron install pre-preparation:

  • On all nodes Install pre-requisites for Ambari:

    Code Block
    yum install git wget curl rpm tar unzip scp bzip2 wget createrepo yum-utils ntp python-pip psutils python-psutil ntp libffi-devel gcc openssl-devel -y
pip install --upgrade pip
pip install requests

...



NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MySQL
      SERVERS IN PRODUCTION USE!  PLEASE READ EACH STEP CAREFULLY!

In order to log into MySQL to secure it, we'll need the current
password for the root user.  If you've just installed MySQL, and
you haven't set the root password yet, the password will be blank,
so you should just press enter here.

Enter current password for root (enter for none):
OK, successfully used password, moving on...

Setting the root password ensures that nobody can log into the MySQL
root user without the proper authorisation.

Set root password? [Y/n]
New password:
Re-enter new password:
Password updated successfully!
Reloading privilege tables..
 ... Success!


By default, a MySQL installation has an anonymous user, allowing anyone
to log into MySQL without having to have a user account created for
them.  This is intended only for testing, and to make the installation
go a bit smoother.  You should remove them before moving into a
production environment.

Remove anonymous users? [Y/n] n
 ... skipping.

Normally, root should only be allowed to connect from 'localhost'.  This
ensures that someone cannot guess at the root password from the network.

Disallow root login remotely? [Y/n]
 ... Success!
By default, MySQL comes with a database named 'test' that anyone can
access.  This is also intended only for testing, and should be removed
before moving into a production environment.

Remove test database and access to it? [Y/n]
 - Dropping test database...
ERROR 1008 (HY000) at line 1: Can't drop database 'test'; database doesn't exist
 ... Failed!  Not critical, keep moving...
 - Removing privileges on test database...
 ... Success!

Reloading the privilege tables will ensure that all changes made so far
will take effect immediately.

Reload privilege tables now? [Y/n]
 ... Success!

All done!  If you've completed all of the above steps, your MySQL
installation should now be secure.

Thanks for using MySQL!


Cleaning up...
#

Build Metron code


Now we are going to start to building Metron. At the time of writing, Metron 0.4.0 was in the final stages of being 1 was released.

  • On the main node, clone the Metron repository:

    Code Block
    git clone https://github.com/apache/metron

...


  • Clone Metron repo and switch to 0.4.

...

  • 1 release

...

  • :

Code Block
git clone https://github.com/apache/metron
cd metron
git checkout Metron_0.4.01

 

  • Build Metron with HDP 2.5 profile:

    Code Block
    cd metron
mvn clean package -DskipTests -T 2C -P HDP-2.5.0.0,mpack
cd metron-deployment/packaging/docker/rpm-docker
mvn clean install -DskipTests -PHDP-2.5.0.0

...

  • Fetch & create logrotate script for Hadoop Services:

    Code Block
    wget -O /etc/logrotate.d/metron-ambari https://raw.githubusercontent.com/apache/metron/master/metron-deployment/roles/ambari_common/templates/metron-hadoop-logrotate.yml
sed -i 's/^  {{ hadoop_logrotate_frequency }}.*$/  daily/' /etc/logrotate.d/metron-ambari
sed -i 's/^  rotate {{ hadoop_logrotate_retention }}.*$/  rotate 30/' /etc/logrotate.d/metron-ambari
chmod 0644 /etc/logrotate.d/metron-ambari

Ambari 2.4 with HDP 2.5 install


Inspired by: [https://docs.hortonworks.com/HDPDocuments/Ambari-2.4.3.0/bk_ambari-installation/content/ch_Getting_Ready.html]

...

# yum install monit -y
# wget -O /etc/monitrc https://github.com/apache/metron/raw/master/metron-deployment/roles/monit/templates/monit/monit.conf
# sed -i 's/{{ inventory_hostname }}/<IP ADDRESS>/' /etc/monitrc
# sed -i 's/{{ monit_user }}/admin/' /etc/monitrc
# sed -i 's/{{ monit_pass }}/monit/' /etc/monitrc
# chmod 600 /etc/monitrc
# wget -O /etc/monit.d/pcap-replay.monit https://github.com/apache/metron/raw/master/metron-deployment/roles/monit/templates/monit/pcap-replay.monit
# chmod 644 /etc/monit.d/pcap-replay.monit
# wget -O /etc/monit.d/pcap-service.monit https://github.com/apache/metron/raw/master/metron-deployment/roles/monit/templates/monit/pcap-service.monit
# chmod 644 /etc/monit.d/pcap-service.monit
# wget -O /etc/monit.d/pycapa.monit https://github.com/apache/metron/raw/master/metron-deployment/roles/monit/templates/monit/pycapa.monit
# chmod 644 /etc/monit.d/pycapa.monit
# wget -O /etc/monit.d/snort.monit https://github.com/apache/metron/raw/master/metron-deployment/roles/monit/templates/monit/snort.monit
# chmod 644 /etc/monit.d/snort.monit
# wget -O /etc/monit.d/yaf.monit https://github.com/apache/metron/raw/master/metron-deployment/roles/monit/templates/monit/yaf.monit
# chmod 644 /etc/monit.d/yaf.monit
# wget -O /etc/monit.d/bro.monit https://github.com/apache/metron/raw/master/metron-deployment/roles/monit/templates/monit/bro.monit
# sed -i 's/^  with pidfile.*$/  with pidfile \/usr\/local\/bro\/spool\/bro\/\.pid/' /etc/monit.d/bro.monit
# chmod 644 /etc/monit.d/bro.monit
# systemctl enable monit
# systemctl start monit
# systemctl status monit
# monit reload
# monit stop all
# monit start all
# monit summary | tail -n +3 | awk -F"'" '{print $2}'

Miscellaneous Issues

  • There's currently a bug in Metron 0.4.0 where Metron REST doesn't start when restarting the Metron services. This bug was fixed in METRON-990 (https://github.com/apache/metron/pull/613/commits/1a9b19a0101ada58cb671ab224934f304df6fff8) but unfortunately, this fix didn't make the 0.4.0 release. In order to fix this, edit the file "/etc/rc.d/init.d/metron-rest" and on line 148, change "$0 start" to "$0 start $2".

  • I had a problem with Zeppelin after rebooting this machine and had to manually create the Zeppelin run directory:

...

This was fixed by deleting the Kibana index ".kibana": "curl -XDELETE http://localhost:9200/.kibana"

Miscellaneous Services

 

  • Load the correct Elasticsearch template with:
    "# curl -s -w "%{http_code}" -u <USERNAME>:<PASSWORD> -H "X-Requested-By: ambari" -X POST -d '{ "RequestInfo": { "context": "Install ES Template from REST", "command": "ELASTICSEARCH_TEMPLATE_INSTALL"},"Requests/resource_filters": [{"service_name": "METRON","component_name": "METRON_INDEXING","hosts" : "<HOSTNAME>"}]}' http://<AMBARI HOST>:8080/api/v1/clusters/<CLUSTERNAME>/requests"

...

# nifi-1.2.0/bin/nifi.sh install
# nifi-1.2.0/bin/nifi.sh start

Exposed Interfaces


In the end, you'll end up with a bunch of exposed UIs:
- Ambari: http://node1:8080/
- Kibana: http://node1:5000/
- Sensor Status (monit): http://node1:2812
- Elasticsearch: http://node1:9200/
- Storm UI: http://node1:8744/
- Metron REST interface: http://node1:8082/swagger-ui.html#/
- Management UI: http://node1:4200/ (user/password)
- Apache Nifi: http://node1:8089/nifi/
- Zookeeper: http://node1:2181
- Kafka: http://node1:6667


TROUBLESHOOTING