Page History

...

In this configuration any linux user template can be used to spawn elasticsearch nodes. The number of such nodes should be configurable via a global parameter. One of the node will be designated as the master node, which will also run the redis instance.

Config for example elasticsearch cluster:

This will help configuring unicast discovery of master nodes. Multicast discovery can also be used but is not described here. By default all nodes are enabled to function as master. The actual master is elected through a elasticsearch master election process.

On each elasticsearch node:

• Edit the file /etc/elasticsearch/elasticsearch.yml, replace host1, host2 and so on with the actual ip address of the nodes.

discovery.zen.ping.unicast.hosts: ["host1", "host2"]

• Edit the file /etc/elasticsearch/elasticsearch.yml, replace host1 with the nodes ip address

network.publish_host: host1

Logstash Configuration on the log shipping layer.

...

input {
  redis {
    host => "<host>"
    # these settings should match the output of the agent
    data_type => "list"
    key => "logstash"
    # We use the 'json' codec here because we expect to read
    # json events from redis.
    codec => json
  }
}
filter
{
        grok
        {
                match => [ "message","%{YEAR:year}-%{MONTHNUM:month}-%{MONTHDAY}[T ]%{HOUR}:?%{MINUTE}:?%{SECOND}[T ]INFO%{GREEDYDATA}job[\-]+%{INT:jobid}\s*=\s*\[\s*%{UUID:uuid}\s*\]%{GREEDYDATA}"]
                named_captures_only => true
        }
}
output
{
  stdout { debug => true debug_format => "json"}
  elasticsearch {
        host => "<host>"
  }
}

Steps in setting up the service:

...