THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
Adjust limits to secure level ([link](https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_installing_manually_book/content/ref-729d1fb0-6d1b-459f-a18a-b5eba4540ab5.1.html)):
Code Block ulimit -n 32768 ulimit -u 65536 echo -e "* - nofile 32768\n* - nproc 65536" >> /etc/security/limits.conf
Disable IPv6, leaving it enabled may force service to bind to IPv6 addresses only and thus resulting in inability to connect to it ([source link](https://wiki.centos.org/FAQ/CentOS6#head-d47139912868bcb9d754441ecb6a8a10d41781df)):
Code Block sysctl -w net.ipv6.conf.all.disable_ipv6=1 sysctl -w net.ipv6.conf.default.disable_ipv6=1 echo -e "\n# Disable IPv6\nnet.ipv6.conf.all.disable_ipv6 = 1\nnet.ipv6.conf.default.disable_ipv6 = 1" >> /etc/sysctl.conf
Disable Transparent Hugepage. Add "transparent_hugepage=never" to the end of the kernel line in "/etc/default/grub" and reboot. (Ambari demands it, do we need to comply?)
Code Block # Change the line: GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=cl/root rd.lvm.lv=cl/swap rhgb quiet" # To: GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=cl/root rd.lvm.lv=cl/swap rhgb quiet transparent_hugepage=never" # Afterwards, run: grub2-mkconfig -o /boot/grub2/grub.cfg
After reboot check that changes were applied (make sure that word "never" is selected in square-brackets):
Code Block cat /sys/kernel/mm/transparent_hugepage/enabled always madvise [never]
Alternatively, if you do not want to mess with kernel parameters, you can create a new systemd service which disables this on each boot. Create the file "/etc/systemd/system/disable-thp.service" with the following content:
Code Block [Unit] Description=Disable Transparent Huge Pages (THP) [Service] Type=simple ExecStart=/bin/sh -c "echo 'never' > /sys/kernel/mm/transparent_hugepage/enabled && echo 'never' > /sys/kernel/mm/transparent_hugepage/defrag" [Install] WantedBy=multi-user.target
Restart systemd, start the new service and make sure the new service runs at startup:
Code Block # systemctl daemon-reload # systemctl start disable-thp # systemctl enable disable-thp
Disable SELinux (is a must to install Ambari and build Metron:
Code Block setenforce 0 sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
Metron install pre-preparation
On all nodes Install pre-requisites for Ambari:
Code Block yum install git wget curl rpm tar unzip scp bzip2 wget createrepo yum-utils ntp python-pip psutils python-psutil ntp libffi-devel gcc openssl-devel -y pip install --upgrade pip pip install requests
...
Check whether Maven works:
Code Block source /etc/profile.d/maven.sh mvn -V
You should see something similar to:
...
Code Block # mvn -V
...
Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T08:41:47-08:00)
...
Maven home: /opt/apache-maven-3.3.9
...
Java version: 1.8.0_131, vendor: Oracle Corporation
...
Java home: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.131-3.b12.el7_3.x86_64/jre
...
Default locale: en_US, platform encoding: UTF-8
...
OS name: "linux", version: "3.10.0-514.16.1.el7.x86_64", arch: "amd64", family: "unix"
...
[INFO] Scanning for projects...
...
[INFO] ------------------------------------------------------------------------
...
[INFO] BUILD FAILURE
...
[INFO] ------------------------------------------------------------------------
...
[INFO] Total time: 0.083 s
...
[INFO] Finished at: 2017-06-06T09:59:03-07:00
...
[INFO] Final Memory: 13M/479M
...
[INFO] ------------------------------------------------------------------------
...
[ERROR] No goals have been specified for this build. You must specify a valid lifecycle phase or a goal in the format <plugin-prefix>:<goal> or <plugin-group-id>:<plugin-artifact-id>[:<plugin-version>]:<goal>. Available lifecycle phases are: validate, initialize, generate-sources, process-sources, generate-resources, process-resources, compile, process-classes, generate-test-sources, process-test-sources, generate-test-resources, process-test-resources, test-compile, process-test-classes, test, prepare-package, package, pre-integration-test, integration-test, post-integration-test, verify, install, deploy, pre-clean, clean, post-clean, pre-site, site, post-site, site-deploy. -> [Help 1]
...
[ERROR]
...
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
...
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
...
[ERROR]
...
[ERROR] For more information about the errors and possible solutions, please read the following articles:
...
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/NoGoalSpecifiedException
...
On Ambari node install and enable docker (we will need it to build Metron mpack for Ambari):
Code Block yum install docker-io -y systemctl start docker
...
If you haven't run `mysql_secure_installation` after the database installation, do that first:
| Code Block |
|---|
systemctl start mariadb
systemctl enable mariadb
systemctl status mariadb
mysql_secure_installation
|
Should produce following output:
| Code Block |
|---|
NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MySQL |
...
SERVERS IN PRODUCTION USE! |
...
PLEASE READ EACH STEP CAREFULLY! |
...
In order to log into MySQL to secure it, we'll need the current |
...
password for the root user. |
...
If you've just installed MySQL, and |
...
you haven't set the root password yet, the password will be blank, |
...
so you should just press enter here. |
...
Enter current password for root (enter for none): |
...
OK, successfully used password, moving on... |
...
Setting the root password ensures that nobody can log into the MySQL |
...
root user without the proper authorisation. |
...
Set root password? [Y/n] |
...
New password: |
...
Re-enter new password: |
...
Password updated successfully! |
...
Reloading privilege tables.. |
...
... Success! |
...
By default, a MySQL installation has an anonymous user, allowing anyone |
...
to log into MySQL without having to have a user account created for |
...
them. |
...
This is intended only for testing, and to make the installation |
...
go a bit smoother. |
...
You should remove them before moving into a |
...
production environment. |
...
Remove anonymous users? [Y/n] n |
...
... skipping. |
...
Normally, root should only be allowed to connect from 'localhost'. |
...
This |
...
ensures that someone cannot guess at the root password from the network. |
...
Disallow root login remotely? [Y/n] |
...
... Success! |
...
By default, MySQL comes with a database named 'test' that anyone can |
...
access. |
...
This is also intended only for testing, and should be removed |
...
before moving into a production environment. |
...
Remove test database and access to it? [Y/n] |
...
- Dropping test database... |
...
ERROR 1008 (HY000) at line 1: Can't drop database 'test'; database doesn't exist |
...
... Failed! |
...
Not critical, keep moving... |
...
- Removing privileges on test database... |
...
... Success! |
...
Reloading the privilege tables will ensure that all changes made so far |
...
will take effect immediately. |
...
Reload privilege tables now? [Y/n] |
...
... Success! |
...
All done! |
...
If you've completed all of the above steps, your MySQL |
...
installation should now be secure. |
...
Thanks for using MySQL! |
...
Cleaning up... |
...
Build Metron code
Now we are going to start to building Metron . At the time of writing, Metron 0.4.1 was released.code:
Clone Metron repo and switch to 0.4.1 release:
Code Block git clone https://github.com/apache/metron cd metron git checkout Metron_0.4.1
...
Build Metron with HDP 2.5 profile:
Code Block cd metron mvn clean package -DskipTests -T 2C -P HDP-2.5.0.0,mpack cd metron-deployment/packaging/docker/rpm-docker mvn clean install -DskipTests -PHDP-2.5.0.0
If for some reason, the rpm-docker fails with the message "/bin/bash: ./build.sh: Permission denied", try disabling selinux:
| Code Block |
|---|
setenforce 0 |
And run "mvn clean install -DskipTests -PHDP-2.5.0.0" mvn commands again.
On all nodes, create a localrepo directory and copy the RPMs from Ambari node there:
Code Block mkdir /localrepo cp -rp /root/metron/metron-deployment/packaging/docker/rpm-docker/RPMS/noarch/* /localrepo/ createrepo /localrepo
If you're doing a multi node install, also copy the packages to the other nodes:
Code Block ssh root@node2 mkdir /localrepo scp /localrepo/*rpm root@node2:/localrepo/ ssh root@node2 createrepo /localrepo
Make sure to do the above on each node.
Fetch & create logrotate script for Hadoop Services:
Code Block wget -O /etc/logrotate.d/metron-ambari https://raw.githubusercontent.com/apache/metron/master/metron-deployment/roles/ambari_common/templates/metron-hadoop-logrotate.yml sed -i 's/^ {{ hadoop_logrotate_frequency }}.*$/ daily/' /etc/logrotate.d/metron-ambari sed -i 's/^ rotate {{ hadoop_logrotate_retention }}.*$/ rotate 30/' /etc/logrotate.d/metron-ambari chmod 0644 /etc/logrotate.d/metron-ambari
Ambari 2.4 with HDP 2.5 install
Inspired by: [https://docs.hortonworks.com/HDPDocuments/Ambari-2.4.3.0/bk_ambari-installation/content/ch_Getting_Ready.html]
Enable time sync, disable firewall and SElinux on every node:
Code Block systemctl enable ntpd systemctl start ntpd systemctl stop firewalld systemctl disable firewalld setenforce 0 (=> I know, but for the sake of simplicity, quickness & testing, I've disabled selinux.)
Make sure each node can resolve every other node's hostname or add hostname of each node to `/etc/hosts` on every node. For example add following lines in /etc/hosts of each node:
Code Block 10.10.10.1 node1
...
10.10.10.2 node2
...
10.10.10.3 node3 10.10.10.4 node4
Where 10.10.10.1, 10.10.10.2, 10.10.10.3 and 10.10.10.
...
4 are the IP addresses of your nodes and "node1", "node2", "node3" and "
...
node4" are their respective hostnames.
On main node download and setup Ambari repo (you may replace the "2.4.23.0" with a newer Ambari version number):
Code Block wget -nv http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.4.3.0/ambari.repo -O /etc/yum.repos.d/ambari.repo
...