Page History

...

IMPORTANT: Some of the values specified in this article need to be substituted with your info, like IP addresses and hostnames. Such values are enclosed with brackers <like that>. Please make sure replace them with your info. 

Prerequisites:

• Centos 6

• In order to ease install make sure that your main interface is named eth0. If it is not, you need to adjust some Ambari services configuration accordingly (e.g. ElasticSearch)

• Single-node: At least 48 GB RAM, 8 cores and 400 GB HDD. Multi-node: At least 32 GB RAM, 4 cores and 200 GB HDD for smooth performance.

• (optional) Disable PackageKit if it is installed, if it is not, just ignore this point:

  Code Block
  sed -i 's/enabled=1/enabled=0/g' /etc/yum/pluginconf.d/refresh-packagekit.conf

...

• Disable Transparent Hugepage. Add "transparent_hugepage=never" to the end of the kernel line in grub.conf and reboot. (Ambari demands it, do we need to comply?):

  Code Block
  transparent_hugepage=never

  After reboot check that changes were applied (make sure that word "never" is selected in square-brackets):

  Code Block
  # cat /sys/kernel/mm/transparent_hugepage/enabled always madvise [never]

Metron install pre-preparation

On all nodes Install pre-requisites for Ambari:

yum install epel-release -y
yum update -y
yum install git wget curl rpm scp tar unzip bzip2 wget createrepo reposync yum-utils ntp python-pip -y

...

yum install docker-io -y
service docker start

yum install npm -y

Build Metron code:

On main node clone Metron repository:

...

If passwordless ssh has not yet been set up within the cluster, then in main node generate key and add it to trusted:

cat /dev/zero | ssh-keygen -q -N "" 2>/dev/null
ssh-copy-id -i ~/.ssh/id_rsa.pub <replace_with_master_node_ip>
chmod 700 ~/.ssh

Add this key to Add this key to all the slave nodes:

ssh-copy-id -i ~/.ssh/id_rsa.pub <replace_with_node_ip>

Ambari 2.4 with HDP 2.5 install

Inspired by: http://docs.hortonworks.com/HDPDocuments/Ambari-2.4.1.0/bk_ambari-installation/content/ch_Getting_Ready.html

On all nodes disable security:

Adjust limits to secure level (inspired by link):

ulimit -n 

...

32768
ulimit -u 65536
echo -e "* - nofile 32768\n* - nproc 65536" >> /etc/security/limits.conf

Enable time sync, disable firewall and SElinux:

257597
chkconfig ntpd on
service ntpd start
chkconfig iptables off
/etc/init.d/iptables stop
setenforce 0

...

wget -nv http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.4.13.0/ambari.repo -O /etc/yum.repos.d/ambari.repo

...

# yum repolist | grep HDPambari
HDPUpdates-ambari-2.4.1.5                  HDP-2.5                                             200
HDP-UTILS-1.1.0.21       HDP-UTILS-1.1.0.21                                   51

Install and setup Ambari server:

yum install ambari-server -y
ambari-server setup -s

Add Metron service to Ambari by running mpack command (make sure to specify correct path to mpack in --mpack=):

ambari-server install-mpack --mpack=incubator-metron/metron-deployment/packaging/ambari/metron-mpack/target/metron_mpack-1.0.0.0-SNAPSHOT.tar.gz --verbose

Start Ambari:

ambari-server start

Access the Ambari UI by going to the following URL in a web browser (use admin / admin as user / pass):

http://<replace_with_master_node_ip>:8080/#/installer/step0

Get Started page: Enter any desired cluster name.

Select Version: Make sure "Public Repository" is checked.

Install Options: Specify hostnames of your nodes where Ambari cluster should be installed (all the ones you have specified in /etc/hosts) in Target Hosts. Copy content of the main node private key (/root/.ssh/id_rsa) in "Host Registration Information". If you receive warning like below, ignore it and click OK:

The following hostnames are not valid FQDNs

Choose Services: Select following Services:

0 ambari-2.4.1.0 - Updates

Install and setup Ambari server:

yum install ambari-server -y
ambari-server setup -s

Add Metron service to Ambari by running mpack command (make sure to specify correct path to mpack in --mpack=):

ambari-server install-mpack --mpack=incubator-metron/metron-deployment/packaging/ambari/metron-mpack/target/metron_mpack-1.0.0.0-SNAPSHOT.tar.gz --verbose

Start Ambari:

ambari-server start

Access the Ambari UI by going to the following URL in a web browser (use admin / admin as user / pass):

http://<replace_with_master_node_ip>:8080/#/installer/step0

Get Started page: Enter any desired cluster name.

Select Version: Make sure "Public Repository" is checked.

Install Options: Specify hostnames of your nodes where Ambari cluster should be installed (all the ones you have specified in /etc/hosts) in Target Hosts. Copy content of the main node private key (/root/.ssh/id_rsa) in "Host Registration Information". If you receive warning like below, ignore it and click OK:

The following hostnames are not valid FQDNs

Choose Services: Select following Services:

Assign Masters: Assign "Kafka Broker" on all nodes. Make sure move Assign Masters: Assign "Kafka Broker" on all nodes. Make sure move following components on one common node:

...

Assign Slaves and Clients: select All forand Clients: select All for:

Customize Services: Following are the list of service that need to be configured:

Customize Services: Following are the list of service that need to be configured:

Set rest of the configuration values to recommended by Ambari or the one you desire (like DB passwords) and perform install.

After install configuration

Fix ElasticSearch permission (it will crash right after start in Ambari) (unless it is already fixed as reported in METRON-642):

chown -Rh elasticsearch:

...

"es.ip": "{{ es_url }}",

to:

"es.ip": "<replace_with_elasticsearch_master_hostname>",

"es.port": "9300", 

elasticsearch /etc/elasticsearch

As it will be owned by root by default and will drop error:

Fix path to ES log file in Java parameter 

Set rest of the configuration values to recommended by Ambari or the one you desire (like DB passwords) and perform install.

After install configuration

Fix ElasticSearch permission (it will crash right after start in Ambari) (unless it is already fixed as reported in METRON-642):

chown -Rh elasticsearch:elasticsearch /etc/elasticsearch

As it will be owned by root by default and will drop error:

Fix path to ES log file in Java parameter (unless it is already fixed as reported in METRON-642)::

sed -i 's@elasticsearchelasticsearch@elasticsearch/elasticsearch@g' /etc/sysconfig/elasticsearch

It is ok if some service will not able to start, check the errors and start them all manually.

...

METRON-642)::

sed -i 's@elasticsearchelasticsearch@elasticsearch/elasticsearch@g' /etc/sysconfig/elasticsearch

It is ok if some service will not able to start, check the errors and start them all manually.

TROUBLESHOOTING

Ignore the error Storm UI shown on the screenshot below if you've built your Metron code with HDP-2.5.0.0 profile (in Maven):

Image Added

It appears because your Kafka Topic was not created or contains no data. Setup streaming and make sure your Kafka topic (from which topology should read data) exists.

If you have GUI installed on your server you should run following, before running git clone command:

...