Page History

...

• Multiple nodes:
• 3 VMs, 2 CPUs per VM and 8 GB RAM per VM.
• Hosts:
• 10.10.10.1 node1
• 10.10.10.2 node2
• 10.10.10.3 node3

Prerequisites:

• CentOS 7

• Add the epel repository and

  install tmux, vim & htop. Installing these utilities is not strictly necessary, but I install these by default for potential troubleshooting & editing of files locally:

...

• update your system:

  Code Block
  yum install epel-release -y

...

• yum update

...

...

• -y

• Set up passwordless SSH between our nodes

  :

  . If passwordless ssh has not yet been set up within the cluster, then in main node generate key:

  Code Block

...

• cat /dev/zero | ssh-keygen -q -N "" 2>/dev/null

...

• If you're not installing on a single node, add this newly generated key to all the slave nodes:

  Code Block
  ssh-copy-id -i ~/.ssh/id_rsa.pub <replace_with_node_ip>

Side note: You might have to adapt your sshd_config file and add "PermitRootLogin yes" amongst other parameters if you want passwordless root access, but that's outside the scope of this document.

• Increase limits for ElasticSearch and Storm on nodes where you will be installing them (if you don't know, increase it everywhere):

...

• Code Block
  echo -e "elasticsearch - memlock unlimited\nstorm - nproc 257597" >> /etc/security/limits.conf

• Adjust limits to secure level ([link](https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_installing_manually_book/content/ref-729d1fb0-6d1b-459f-a18a-b5eba4540ab5.1.html)):

  Code Block

...

• ulimit -n 32768

...

• ulimit -u 65536

...

• echo -e "* - nofile 32768\n* - nproc 65536" >> /etc/security/limits.conf

• Disable IPv6, leaving it enabled may force service to bind to IPv6 addresses only and thus resulting in inability to connect to it ([source link](https://wiki.centos.org/FAQ/CentOS6#head-d47139912868bcb9d754441ecb6a8a10d41781df)):
  Disable for the running system:

...

sysctl -w net.ipv6.conf.all.disable_ipv6=1

...

sysctl -w net.ipv6.conf.default.disable_ipv6=1

...

echo 

...

-e "\n# Disable IPv6\nnet.ipv6.conf.all.disable_ipv6 =

...

 1\nnet.ipv6.conf.default.disable_ipv6 = 1"

...

 >> /etc/sysctl.conf

...

• Disable Transparent Hugepage. Add "transparent_hugepage=never" to the end of the kernel line in "/etc/default/grub" and reboot. (Ambari demands it, do we need to comply?)

  Code Block
  # Change the line:

...

• GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=cl/root rd.lvm.lv=cl/swap rhgb quiet"

...

• # To:

...

• GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=cl/root rd.lvm.lv=cl/swap rhgb quiet transparent_hugepage=never"

...

• # Afterwards, run:

...

• grub2-mkconfig -o /boot/grub2/grub.cfg

After reboot check that changes were applied (make sure that word "never" is selected in square-brackets):

...

cat /sys/kernel/mm/transparent_hugepage/enabled

...

always madvise [never]

Alternatively, if you do not want to mess with kernel parameters, you can create a new systemd service which disables this on each boot. Create the file "/etc/systemd/system/disable-thp.service" with the following content:

[Unit]

...

Description=Disable Transparent Huge Pages (THP)

...

[Service]

...

Type=simple

...

ExecStart=/bin/sh -c "echo 'never' > /sys/kernel/mm/transparent_hugepage/enabled && echo 'never' > /sys/kernel/mm/transparent_hugepage/defrag"

...

[Install]

...

WantedBy=multi-user.target

Restart systemd, start the new service and make sure the new service runs at startup:

...

Metron install pre-preparation:

• On all nodes Install pre-requisites for Ambari:

  Code Block

...

• yum install git wget curl rpm tar unzip scp bzip2 wget createrepo yum-utils ntp python-pip psutils python-psutil ntp libffi-devel gcc openssl-devel -y

...

• pip install --upgrade pip

...

• pip install requests

• Install Maven on main node and on

  On Metron node install java 1.8 (if you don't know which it is, install it everywhere):

  Code Block

...

• yum install java-1.8.0-openjdk java-1.8.0-openjdk-devel -y

• Set path to Java 8 if it does not exist:

  Code Block

...

• export JAVA_HOME=$(readlink -f /usr/bin/java | sed "s_/jre/bin/java__")

• Save export for future reboots:

...

• Code Block
  echo 'export JAVA_HOME=$(readlink -f /usr/bin/java | sed "s_/jre/bin/java__")' > /etc/profile.d/java_18.sh

...

• chmod +x /etc/profile.d/java_18.sh

...

• source /etc/profile.d/java_18.sh

• Download and install Maven 3.3.9:

...

• Code Block
  wget http://apache.volia.net/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz

...

• tar -zxf apache-maven-3.3.9-bin.tar.gz

...

• mv apache-maven-3.3.9 /opt

...

• PATH=/opt/apache-maven-3.3.9/bin:$PATH

...

• echo 'export PATH=/opt/apache-maven-3.3.9/bin:$PATH' > /etc/profile.d/maven.sh

...

• chmod +x /etc/profile.d/maven.sh

• Check whether Maven works:

  Code Block

...

• source /etc/profile.d/maven.sh

...

• mvn -V

You should see something similar to:

[root@base1 ~]# mvn -V
Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T08:41:47-08:00)
Maven home: /opt/apache-maven-3.3.9
Java version: 1.8.0_131, vendor: Oracle Corporation
Java home: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.131-3.b12.el7_3.x86_64/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "3.10.0-514.16.1.el7.x86_64", arch: "amd64", family: "unix"
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 0.083 s
[INFO] Finished at: 2017-06-06T09:59:03-07:00
[INFO] Final Memory: 13M/479M
[INFO] ------------------------------------------------------------------------
[ERROR] No goals have been specified for this build. You must specify a valid lifecycle phase or a goal in the format <plugin-prefix>:<goal> or <plugin-group-id>:<plugin-artifact-id>[:<plugin-version>]:<goal>. Available lifecycle phases are: validate, initialize, generate-sources, process-sources, generate-resources, process-resources, compile, process-classes, generate-test-sources, process-test-sources, generate-test-resources, process-test-resources, test-compile, process-test-classes, test, prepare-package, package, pre-integration-test, integration-test, post-integration-test, verify, install, deploy, pre-clean, clean, post-clean, pre-site, site, post-site, site-deploy. -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/NoGoalSpecifiedException
[root@base1 ~]#

• On Ambari node install and enable docker (we will need it to build Metron mpack for Ambari):

  Code Block

...

• yum install docker-io -y

...

• systemctl start docker

• Also on your build box, install npm. This is needed to build metron-config, part of the UI.

  Code Block

...

• yum install npm -y

• Remove ipv4 'localhost.localdomain' from /etc/hosts
  Remove ipv6 'localhost.localdomain' from /etc/hosts
  Add "127.0.0.1    localhost" to /etc/hosts if it does not already there

• Install the database we will use for Metron REST:

  Code Block

...

• yum install mariadb-server mysql-connector-java -y

• Configure a user and a database for Metron REST:

If you haven't run `mysql_secure_installation` after the database installation, do that first:

...

systemctl start mariadb

...

systemctl enable mariadb

...

systemctl status mariadb

...

mysql_secure_installation

NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MySQL
      SERVERS IN PRODUCTION USE!  PLEASE READ EACH STEP CAREFULLY!

In order to log into MySQL to secure it, we'll need the current
password for the root user.  If you've just installed MySQL, and
you haven't set the root password yet, the password will be blank,
so you should just press enter here.

Enter current password for root (enter for none):
OK, successfully used password, moving on...

Setting the root password ensures that nobody can log into the MySQL
root user without the proper authorisation.

Set root password? [Y/n]
New password:
Re-enter new password:
Password updated successfully!
Reloading privilege tables..
 ... Success!


By default, a MySQL installation has an anonymous user, allowing anyone
to log into MySQL without having to have a user account created for
them.  This is intended only for testing, and to make the installation
go a bit smoother.  You should remove them before moving into a
production environment.

Remove anonymous users? [Y/n] n
 ... skipping.

Normally, root should only be allowed to connect from 'localhost'.  This
ensures that someone cannot guess at the root password from the network.

Disallow root login remotely? [Y/n]
 ... Success!
By default, MySQL comes with a database named 'test' that anyone can
access.  This is also intended only for testing, and should be removed
before moving into a production environment.

Remove test database and access to it? [Y/n]
 - Dropping test database...
ERROR 1008 (HY000) at line 1: Can't drop database 'test'; database doesn't exist
 ... Failed!  Not critical, keep moving...
 - Removing privileges on test database...
 ... Success!

Reloading the privilege tables will ensure that all changes made so far
will take effect immediately.

Reload privilege tables now? [Y/n]
 ... Success!

All done!  If you've completed all of the above steps, your MySQL
installation should now be secure.

Thanks for using MySQL!


Cleaning up...
#

...

Now we are going to start to building Metron. At the time of writing, Metron 0.4.0 was in the final stages of being released.

• On the main node, clone the Metron repository:

...

• Code Block
  git clone https://github.com/apache/metron

If you want to make sure you're on the 0.4.0 release branch, do:

...

git clone https://github.com/apache/metron

...

cd metron

...

git checkout Metron_0.4.0

• Build Metron with HDP 2.5 profile:

  Code Block

...

• cd metron

...

• mvn clean package -DskipTests -T 2C -P HDP-2.5.0.0,mpack

...

• cd metron-deployment/packaging/docker/rpm-docker

...

• mvn clean install -DskipTests -PHDP-2.5.0.0

If for some reason, the rpm-docker fails with the message "/bin/bash: ./build.sh: Permission denied", try disabling selinux ("setenforce 0") and :

setenforce 0

And run "mvn clean install -DskipTests -PHDP-2.5.0.0" again.

• On all nodes, create a localrepo directory and copy the RPMs from Ambari node there:

  Code Block

...

• mkdir /localrepo

...

• cp -rp /root/metron/metron-deployment/packaging/docker/rpm-docker/RPMS/noarch/* /localrepo/

...

• createrepo /localrepo

• If you're doing a multi node install, also copy the packages to the other nodes:

...

• Code Block
  ssh root@node2 mkdir /localrepo

...

• scp /localrepo/*\.rpm root@node2:/localrepo/.

...

• ssh root@node2

...

• createrepo

...

• /localrepo

Make sure to do the above on each node.

• Fetch & create logrotate script for Hadoop Services:

...

• Code Block
  wget -O /etc/logrotate.d/metron-ambari https://raw.githubusercontent.com/apache/metron/master/metron-deployment/roles/ambari_common/templates/metron-hadoop-logrotate.yml

...

• sed -i 's/

...

• ^ {{ hadoop_logrotate_frequency }}.*$/

...

• daily/' /etc/logrotate.d/metron-ambari

...

• sed -i 's/

...

• ^ rotate {{ hadoop_logrotate_retention }}.*$/

...

• rotate 30/'

...

• /etc/logrotate.d/metron

...

Ambari 2.4 with HDP 2.5 install

...

• Adjust limits to secure level (inspired by [link](https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_installing_manually_book/content/ref-729d1fb0-6d1b-459f-a18a-b5eba4540ab5.1.html)):

...

• -ambari chmod 0644 /etc/logrotate.d/metron-ambari

Ambari 2.4 with HDP 2.5 install

Inspired by: [http://docs.hortonworks.com/HDPDocuments/Ambari-2.4.1.0/bk_ambari-installation/content/ch_Getting_Ready.html]

• Enable time sync, disable firewall and SElinux on every node:

  Code Block

...

• systemctl enable ntpd

...

• systemctl start ntpd

...

• systemctl stop firewalld

...

• systemctl disable firewalld

...

• setenforce 0 (=> I know, but for the sake of simplicity, quickness & testing, I've disabled selinux.)

• Make sure each node can resolve every other node's hostname or add hostname of each node to `/etc/hosts` on every node. For example add following lines in /etc/hosts of each node:

...

Where 10.10.10.1, 10.10.10.2 and 10.10.10.3 are the IP addresses of your nodes and "node1", "node2" and "node3" are their respective hostnames.

• On main node download and setup Ambari repo (you may replace the "2.4.2.0" with a newer Ambari version number):

  Code Block

...

• wget -nv http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.4.2.0/ambari.repo -O /etc/yum.repos.d/ambari.repo

...

• yum update -y

• Check that it was added:

# yum repolist | grep ambari
Updates-ambari-2.4.2.0   ambari-2.4.2.0 - Updates                             12

...