Page History

...

ssh into $SQUID_HOST

Install and start Squid:

sudo yum install squid
sudo service squid start

With Squid started, look at the different log files that get created:
```
sudo su -
cd /var/log/squid
ls
```
You see that there are three types of logs available: access.log, cache.log, and squid.out. We are interested in access.log because that is the log that records the proxy usage.
Initially the access.log is empty. Let's generate a few entries for the log, then list the new contents of the access.log:. The "-h 127.0.0.1" indicates that the squidclient will only use the IPV4 interface.
squidclient -h 127.0.0.1 squidclient "http://www.aliexpress.com/af/shoes.html?ltype=wholesale&d=y&origin=n&isViewCP=y&catId=0&initiative_id=SB_20160622082445&SearchText=shoes"
squidclient -h 127.0.0.1 "http://www.help.1and1.co.uk/domains-c40986/transfer-domains-c79878"
squidclient -h 127.0.0.1 "http://www.pravda.ru/science/"
squidclient -h 127.0.0.1 "http://www.brightsideofthesun.com/2016/6/25/12027078/anatomy-of-a-deal-phoenix-suns-pick-bender-chriss"
squidclient "https:-h 127.0.0.1 "https://www.microsoftstore.com/store/msusa/en_US/pdp/Microsoft-Band-2-Charging-Stand/productID.329506400"
squidclient "http://www.autonews.com/article/20151115/RETAIL04/311169971/toyota-fj-cruiser-is-scarce-hot-and-high-priced"
squidclient -h 127.0.0.1 "https://tfl.gov.uk/plan-a-journey/"
squidclient -h 127.0.0.1 "https://www.facebook.com/Africa-Bike-Week-1550200608567001/"
squidclient "http:-h 127.0.0.1 "http://www.ebay.com/itm/02-Infiniti-QX4-Rear-spoiler-Air-deflector-Nissan-Pathfinder-/172240020293?fits=Make%3AInfiniti%7CModel%3AQX4&hash=item281a4e2345:g:iMkAAOSwoBtW4Iwx&vxp=mtr"
squidclient -h 127.0.0.1 "http://www.recruit.jp/corporate/english/company/index.html"
squidclient -h 127.0.0.1 "http://www.lada.ru/en/cars/4x4/3dv/about.html"
squidclient -h 127.0.0.1 "http://www.help.1and1.co.uk/domains-c40986/transfer-domains-c79878"
squidclient -h 127.0.0.1 "http://www.aliexpress.com/af/shoes.html?ltype=wholesale&d=y&origin=n&isViewCP=y&catId=0&initiative_id=SB_20160622082445&SearchText=shoes"
In production environments you would configure your users web browsers to point to the proxy server. But for the sake of simplicity of this tutorial, we will use the client that is packaged with the Squid installation. After we use the client to simulate proxy requests, the Squid log entries should look as follows:
```
1467011157.401 415 127.0.0.1 TCP_MISS/200 337891 GEThttp://www.aliexpress.com/af/shoes.html? - DIRECT/207.109.73.154 text/html
1467011158.083 671 127.0.0.1 TCP_MISS/200 41846 GEThttp://www.help.1and1.co.uk/domains-c40986/transfer-domains-c79878 - DIRECT/212.227.34.3 text/html
1467011159.978 1893 127.0.0.1 TCP_MISS/200 153925 GEThttp://www.pravda.ru/science/ - DIRECT/185.103.135.90 text/html
```
Using the Squid log entries, we can determine the format of the log entries which is:
timestamp | time elapsed | remotehost | code/status | bytes | method | URL rfc931 peerstatus/peerhost | type

...

The first thing we need to do is decide if we will be using the Java-based parser or the Grok-based parser for the new telemetry. In this example we will be using the Grok parser. Grok parser is perfect for structured or semi-structured logs that are well understood (check) and telemetries with lower volumes of traffic (check).
Next we need to define the Grok expression for our log. Refer to Grok documentation for additional details. In our case the pattern is:
SQUID_DELIMITED %{NUMBER:timestamp}
%{SPACE:UNWANTED}
.*%{INT:elapsed} %{
SPACE:UNWANTED}%{IPV4:
IP:ip_src_
addr
address} %{WORD:action}/%{NUMBER:code} %{NUMBER:bytes} %{WORD:method} %{NOTSPACE:url}
-
.*%{
WORD:UNWANTED}\/%{IPV4:ip
IP:ip_dst_addr}
If you do not want to include } %{WORD:UNWANTED}\/%{WORD:UNWANTED}Notice that we apply the UNWANTED tag for any part of the message that we don't want included in our in the resulting JSON structure, you can apply the UNWANTED tag to that section.
Finally, notice that we applied the naming convention to the IPV4 field by referencing the following list of field conventions.
The last thing we need to do is to validate the Grok pattern to make sure it's valid. For our test we will be using a free Grok validator called Grok Constructor. A validated Grok expression should look like this:
Now that the Grok pattern has been defined, we need to save it and move it to HDFS.
1. ssh into HOST $HOST_WITH_ENRICHMENT_TAG as root.
2. Create a file called "squid" in the tmp directory and copy the Grok pattern into the file.
```
touch /tmp/squid
```
3. Open the Squid file and add the Grok pattern defined above.
4. Put the Squid file into the directory where Metron stores its Grok parsers. Existing Grok parsers that ship with Metron are staged under /apps/metron/pattern.
  su - hdfs
```
hadoop fs -rm -r /apps/metron/patterns/squid
```
```
hdfs dfs -put /tmp/squid /apps/metron/patterns/
```

...

ssh into Host $HOST_WITH_ENRICHMENT_TAG as root.
Create a Squid Grok parser configuration file at /usr/metron/$METRON_VERSION/config/zookeeper/parsers/squid.json:
```
touch /usr/metron/$METRON_VERSION/config/zookeeper/parsers/squid.json
```
Add the following contents:
```
{
 "parserClassName": "org.apache.metron.parsers.GrokParser",
 "sensorTopic": "squid",
 "parserConfig": {
 "grokPath": "/apps/metron/patterns/squid",
 "patternLabel": "SQUID_DELIMITED",
 "timestampField": "timestamp"
 },
 "fieldTransformations" : [
   {
     "transformation" : "MTLSTELLAR"
    ,"output" : [ "full_hostname", "domain_without_subdomains" ]
    ,"config" : {
                  "full_hostname" : "URL_TO_HOST(url)"
                 ,"domain_without_subdomains" : "DOMAIN_REMOVE_SUBDOMAINS(full_hostname)"
                }
    }
  ]
```
```
 }
```
Notice the use of the fieldTransformations in the parser configuration. Our Grok Parser is set up to extract the URL, but really we want just the domain or even the domain without subdomains. To do this, we can use the Metron Transformation Language field transformation. The Metron Transformation Language is a Domain Specific Language that allows users to define extra transformations to be done on the messages flowing through the topology. It supports a wide range of common network and string-related functions as well as function composition and list operations. In our case, we extract the hostname from the URL via the URL_TO_HOST function and remove the domain names with DOMAIN_REMOVE_SUBDOMAINS thereby creating two new fields, "full_hostname" and "domain_without_subdomains" to each message. Image Modified
4.
All

parser

configurations

are

stored

in

Zookeeper. Use

the

following

script

to

upload

configurations

to

Zookeeper:
```
   /usr/metron/$METRON_VERSION/bin/zk_load_configs.sh --mode PUSH -i /usr/metron/$METRON_VERSION/config/zookeeper -z $ZOOKEEPER_HOST:2181 
```
Note: You might receive the following warning messages when you execute the previous command. You can safely ignore these warning messages.
log4j:WARN No appenders could be found for logger (org.apache.curator.framework.imps.CuratorFrameworkImpl).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

...

more info.

Step 5: Configure Indexing

Next you might want to configure your sensor's indexing. The indexing topology takes the data from a topology that has been enriched and stores the data in one or more supported indices.

You can choose not to configure a sensor's indexing and use the default values. If you leave the writer configuration unspecified, you will see a warning similar to the following in the Storm console: WARNING: Default and (likely) unoptimized writer config used for hdfs writer and sensor squid. You can ignore this warning message if you intend to use the default configuration.

To configure a sensor's indexing:

1. Create a file called squid.json at /usr/metron/$METRON_VERSION/config/zookeeper/indexing/:

touch $METRON_HOME/config/zookeeper/indexing/squid.json

2. Populate it with the following:

  {
    "elasticsearch": {  
       "index": "squid",  
       "batchSize": 5,
   "enabled" : true
    },
    "hdfs":{
 "index": "squid",  
   "batchSize": 5,
       "enabled" : true 
    }

}

This file sets the batch size of 5 and the index name to squid for both the Elasticsearch and HDFS writers.

3. Push the configuration to ZooKeeper:

/usr/metron/$METRON_VERSION/bin/zk_load_configs.sh --mode PUSH -i /usr/metron/$METRON_VERSION/config/zookeeper -z $ZOOKEEPER_HOST:2181

Step 6: Validate the Squid Message

Another thing we can do is validate our messages. Let's say we wanted to make sure that source IPs and destination IPs are valid. The validators are global so we set them up on the global JSON and push them into Zookeeper. The list of available validators can be found here:

ssh into Host $HOST_WITH_ENRICHMENT_TAG as root.
Open up the global validation configuration.
1. Open the /usr/metron/$METRON_VERSION/config/zookeeper/global.json or another text editing tool:

         /vi  vi /usr/metron/$METRON_VERSION/config/zookeeper/global.json

Add the following validation configuration to the file after the es configuration:


  "fieldValidations" : [
   {
      "input" : [ "ip_src_addr", "ip_dst_addr" ],
      "validation" : "IP",
      "config" : {
      "type" : "IPV4"
       }
   }
  ]

Push the global configuration to Zookeeper:

/usr/metron/$METRON_VERSION/bin/zk_load_configs.sh -i /usr/metron/$METRON_VERSION/config/zookeeper -m PUSH -z $ZOOKEEPER_HOST:2181

Dump the configs and validate that were persisted:
```
/usr/metron/$METRON_VERSION/bin/zk_load_configs.sh -m DUMP -z $ZOOKEEPER_HOST:2181
```
Note: You might receive the following warning messages when you execute the previous command. You can safely ignore these warning messages.
log4j:WARN No appenders could be found for logger (org.apache.curator.framework.imps.CuratorFrameworkImpl).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

...

More details on the validation framework can be found in the Validation Framework section: https://github.com/apache/incubator-metron/tree/master/metron-platform/metron-common#transformation-language

Step

...

7: Deploy the new Parser Topology

Now that we have the Squid parser topology defined, let's deploy it to our cluster.

...

The following steps show how to install NiFi. Perform the following as root:

ssh into HOST $NIFI_HOST as root.

Download NiFi.

cd /usr/lib
wget  http://public-repo-1.hortonworks.com/HDF/centos6/1.x/updates/1.2.0.0/HDF-1.2.0.0-91.tar.gz
tar -zxvf HDF-1.2.0.0-91.tar.gz

Edit the NiFi configuration to update the port of the NiFi web app: nifi.web.http.port=8089
```
cd HDF-1.2.0.0/nifi
vi  conf/nifi.properties
//update nifi.web.http.port to 8089
```
Install NiFi as service.
```
bin/nifi.sh install nifi
```
Start the NiFi Service.
```
service nifi start
```
Go to the NiFi Web: http://$NIFI_HOST:8089/nifi/.
Note: Be sure to substitute your NiFi host name for $NIFI_HOST in the url above. If you simply click on the host, the url will specify Node1 which will not work.

Create a NiFi Flow to Stream Events to Metron

...

Drag a processor to the canvas (do this by the dragging the processor icon which is the first icon on the toolbar).
Select the TailFile type of processor, then select Add.
Right click on the processor and select Configure to display the Configure Processor dialog box. In the Settings tab change the name to "Ingest Squid Events"
1. In the Properties tab, configure the following:
Drag another processor to the canvas.
Select the PutKafka type of processor, then select Add.
Right click on the processor and select Configure.
In the Settings tab, change the name to "Stream to Metron,” then click the relationship checkboxes for failure and success.
In the Properties tab, set the following three properties:
1. Known Brokers: $KAFKA_HOST:6667
2. Topic Name: squid
3. Client Name: nifi-squid
Create a connection by dragging the arrow from the Ingest Squid Events processor to the Stream to Metron processor.
Press the Shift key and select the entire flow, then click the play button (green arrow). You should see all of the processor icons turn into green arrows like below:
Generate some data using squidclient (do this for about 20+ sites).
```
squidclient -h 127.0.0.1 "http://www.cnn.com"
```
You should see metrics on the processor of data being pushed into Metron.
Look at the Storm UI for the parser topology and you should see tuples coming in.
After about 5 minutes, you should see a new Elastic Search index called squid_index* in the Elastic Admin UI.

...

ssh into Host $SEARCH_HOST.

Install the head plugin:

usr/share/elasticsearch/bin/plugin -install mobz/elasticsearch-head/1.x

Navigate to elastic head UI: http://SEARCH_HOST:9200/_plugin/head/.
Click on the Browser tab and select squid doc in the left panel, then select one of the sample docs. You should see something like the following:

...

Space shortcuts

Page tree

Versions Compared

Old Version 44

New Version Current

Key

Step 5: Configure Indexing

Step 6: Validate the Squid Message

Step

7: Deploy the new Parser Topology

Create a NiFi Flow to Stream Events to Metron