The following provides instructions on how to install Metron on an existing Ambari managed HDP 2.4 cluster. These are not recommendations for production deployment but rather for dev/testing environments. 

Prerequisites

For the purposes of this discussion, there will be 4 types of server roles:

1. Hadoop Master
2. Hadoop Slave
3. Ambari Server
4. Metron Installer
5. Metron Search
6. Metron Probe
7. Metron Web

Specs of Existing Hadoop Cluster

The following are the specs for Hadoop related nodes

1. All cluster nodes must be running CentOS 6.x
2. The cluster must be an HDP 2.4 managed by Ambari 2.2.X

3. At a minimum, there needs to be at least 2 Hadoop Master nodes and 4 Hadoop slave nodes

4. The minimum spec for each of the Hadoop Slave and Master nodes should be: TODO: Need to add specs.
5. The following services must be installed across the Hadoop Master and Slave nodes
   1. 
      

6. Each of the following components must be installed on at least 4 Hadoop Slave Nodes

   1. 
7. No other workloads should be running on the cluster. In other words, this cluster should be dedicated for Metron jobs.

Specs for Metron Nodes

1. At least 3 nodes must be dedicated for Metron specific components
2. You should have root access on all Metron nodes

3. All Metron nodes (Search, Probes and Web) must have the following:

   1. Centos 6.7
   2. Oracle JDK 1.8

4. The Metron Installer node must have Docker installed (Note: if using Centos, 7.x is preferred for the Metron Installer node)

5. The Metron Installer node requires SSH access to all Hadoop Cluster and Metron nodes. Passwordless is preferred.

Sample Deployment Architecture

Based on the above specs, here is sample deployment architecture..

Install Instructions

Setup

For the below instructions, set the following environmental variables. Plugin in the necessary values for your deployment.

export METRON_HOME=/root/metron/incubator-metron-Metron_0.2.0BETA_rc2
export METRON_0_2_RELEASE_BINARIES=https://dist.apache.org/repos/dist/dev/incubator/metron/0.2.0BETA-RC2-incubating/apache-metron-0.2.0BETA-RC2-incubating.tar.gz
export INSTALL_CONF_DIRECTORY=my_install

Install or Validate existing Hadoop Cluster

1. Install a new HDP 2.4 using the following instructions or ensure you have an existing HDP cluster that follows the specs outlined in the pre-requisites section. 

Download and Build Metron

1. Log into the Metron Installer node
2. Verify Docker is installed and running. 
3. Ensure that this node has ssh access to each of the Metron and Hadoop/HDP nodes. 
4. Download the Metron Binaries from $METRON_0_2_RELEASE_BINARIES
5. Untar it: tar -zxvf apache-metron-0.2.0BETA-RC2-incubating.tar.gz
6. cd $METRON_HOME
7. Run Docker installer image:
    docker run -it -v $METRON_HOME:/root/incubator-metron dlyle65535/metron-ansible:2.0.0.2 bash
8. Build Metron
   cd /root/incubator-metron/
   mvn clean package -DskipTests 

Configure Metron for Install

1. cd /root/incubator-metron/metron-deployment/inventory
2. create a directory called $INSTALL_CONF_DIRECTORY that will hold all of your metron deployment configuration
3. Copy the sample configs into the new directory:
   1. cp - R /root/incubator-metron/metron-deployment/inventory/metron-example/* /root/incubator-metron/metron-deployment/inventory/$INSTALL_CONF_DIRECTORY
4. Setup your inventory hosts by editing the file $METRON_HOME/metron-deployment/inventory/$INSTALL_CONF_DIRECTORY/hosts

   1. Configure with your cluster details by defining which hosts the  Metron services will be installed on by updating these groups:

      [ambari_master] - host running Ambari
      [ambari_slaves] - all Ambari-managed hosts
      [metron_hbase_tables] - host used to create the HBase tables required by Metron. Requires a HBase client.
      [metron_kafka_topics] - host used to create the Kafka topics required by Metron. Requires a Kafka broker.
      [enrichment] - submits the topology code to Storm and requires a Storm client
      [search] - host(s) where Elasticsearch will be installed
      [web] - host where the Metron UI and underlying services will be installed
      [sensors] - host where network data will be collected and published to Kafka

       

   2. Setup your environment configuration by editing the file $METRON_HOME/metron-deployment/inventory/$INSTALL_CONF_DIRECTORY/group_vars/all

Commonly Used Settings:

Ansible

• • • • ansible_ssh_private_key_file: /Path/to/private/key/file Point to the private key file for ssh user on the target hosts
      • ansible_ssh_user: root The name of the ssh user on the target hosts (requires sudo)

Ambari

• • • • ambari_port: 8080 Change if your Ambari instance uses a non-default port
      • ambari_user: admin Change to user on your Ambari instance
      • ambari_password: admin Change to password for your Ambari user above

Kafka

• • • • num_partitions: 3 Change to your desired number of partitions
      • retention_in_gb: 25 Change to your desired retention size

Metron

• • • • java_home: /usr/jdk64/jdk1.8.0_40 Location of Java on all hosts

Sensors

• • • • sensor_test_mode: True Change to false if not running traffic replay
      • sniff_interface: eth0 Interface that the Metron sensors will sniff on the [sensors] host

Search

• • • • elasticsearch_network_interface: eth0  Bind interface for the Elasticsearch host(s)

Install/Deploy Metron to the Cluster

Note: adding -u user -k to the below command lines allows deployment in environments where passwordless SSH has not been enabled.

1. cd /root/incubator-metron/metron-deployment/playbooks
2. Run the following ansible command to deploy Metron with the standard services (Metron Topologies, WebUI and Monit)
   ansible-playbook -i /root/incubator-metron/metron-deployment/inventory/$INSTALL_CONF_DIRECTORY metron_install.yml --skip-tags="solr"
3. Run the following Ansible command to deploy Metron without installing Sensors
    ansible-playbook -i /root/incubator-metron/metron-deployment/inventory/$INSTALL_CONF_DIRECTORY metron_install.yml --skip-tags="solr,sensors"