https://issues.apache.org/jira/browse/CLOUDSTACK-3471
master
As of cloudstack 4.3 there is no api that can aggregate log messages by the job id. Since logging is a typically a service that can be and is usually available outside the IaaS core, it was felt that instead of integrating this as an api within cloudstack, it would be better to provide a generic logsearch service that could be invoked by cloudstack to retrieve log messages. We describe the mechanism on how to achieve this in what follows.
Purpose
This is typically useful to root admin users.
Author | Description | Date |
---|---|---|
Saurav Lahiri | Inital Draft | 12/14/2013 |
Root Admin users can query this service to quickly identify the sequence of steps related to a particular Job.
QA can use this service to link the log messages related to automated test failures.
The system will comprise of a log shipping layer. This layer will be responsible for collecting logs from each of the management server shipping them to a centralized place. We describe how logstash can be used as the shipping layer. It will be configured to use redis to ship individual log files to a centralized location. Fluentd could be another option.
The shipping phase will interact with another layer called the indexer/search layer. This layer will also enable storing the logs in a format that will help in writing search queries. Here we describe the use of logstash to recieve the individual log files and elasticsearch to search through these. Before logstash outputs the recieved messages to elasticsearch, it will apply a specific grok filter that will split the input messages into key value pairs. The key value pair will allow creation of search queries by (key,value). Via the elasticsearch REST api , search queries can be constructed for required jobid.
Instances of Logstash:
Logstash can aggregate log messages from multiple nodes and multiple log files. In a typical production environment, cloudstack is configured with multiple management server instances for scalability and redundancy. One instance of logstash will be configured to run on each of the management server and will ship the log to redis. The logstash process is reasonably light in terms of memory consumption and should not impact the management server.
Instances of elasticsearch and redis
Elasticsearch runs as a horizontal scale out cluster. We describe the process of creating and using seperate Elastic search nodes:
In this configuration any linux user template can be used to spawn elasticsearch nodes. The number of such nodes should be configurable via a global parameter. One of the node will be designated as the master node, which will also run the redis instance.
Using systemvm for elasticsearch nodes:
Currently cloudstack does not allow deployment of default system vms. The only supported types are (virtual router, secondary storage, consoleproxy, internal and external load balancer). These specifc types are handle in their specific Manager code. To enable systemvms to be started by admin, Default System VM manager and VO class will require to be implemented.
Logstash Configuration on the log shipping layer.
input {
file {
type => "apache"
path => [ "/var/log/cloudstack/management/management-server.log" ]
}
}
output {
amqp {
host => "myamqpserver"
exchange_type => "fanout"
name => "rawlogs"
}
}
Logstash configuration on the index/search layer.
input {
amqp {
type => "all"
host => "<host>"
exchange => "rawlogs"
name => "<name>"
}
}
filter {
grok {
type => "apache"
pattern => "%{YEAR}%{MONTHNUM}%{MONTHDAY}[T ]%{HOUR}\:?%{MINUTE}\:?%{SECOND}[T ]INFO%{GREEDYDATA}job[-]+%{INT\:jobid}\s*=\s*[\s*%{UUID\:uuid}\s*
]%{GREEDYDATA}"
}
}
output {
elasticsearch {
host => "<elasticsearch_master>"
}
}
A new API command ExtractLogByJobIdCmd will be introduced. This will be implemented as a synchronous command.
The manager class will implement the actual functionality of querying elastic search for log messages that match the specified filters. For doing this the Elasticsearch REST api queries will be used. Post method will be used with elasticsearch DSL to specify the required query. DSL is quite flexible and in future if support is required to filter by time stamp and other values DSL would help achieve that with ease.
DSL query for searching logs by jobid
{
"query": {
"query_string": {
"query": "\<jobid\>",
"fields" : "jobid"
}
}
}
A new API will introduced which can be accessed as
http://<host>:8080/client/api?command=extractLogByJobId&jobid=<jobid>
None