Page History

...

As of cloudstack 4.3 there is no api that can aggregate log messages by the job id. An api to extract logs by the jobid would make it easier to identify the sequence of steps that have been executed to complete a particular job. In case of failures it would aid in quickly identifying the associated commands/steps that have resulted in the failure.of cloudstack 4.3 there is no api that can aggregate log messages by the job id. An api to extract logs by the jobid would make it easier to identify the sequence of steps that have been executed to complete a particular job. In case of failures it would aid in quickly identifying the associated commands/steps that have resulted in the failure.

Purpose

Since logging is a typically a service that can be and is usually available outside the IaaS core, it was felt that instead of integrating this as an api within cloudstack, it would be better to provide a generic logsearch service that could be invoked by cloudstack to retrieve log messages. We describe the mechanism on how to achieve this in what follows.

Purpose

This is typically useful to root admin usersIn terms of the functionality available to end users, this will provide a cloudstack api called extractLogsByJobid() which will be available only as a ROOT admin API.

References

Document History

...

Feature Specifications

Use cases

1. Root Admin users can

   use

   query this

   API

   service to quickly identify the sequence of steps related to a particular Job.

2. QA can use this

   api

   service to link the log messages related to automated test failures.

Architecture and Design description

The system will comprise of a log shipping layer. This layer will be responsible for collecting logs from each of the management server shipping them to a centralized a centralized place. In the current design we are proposing logstash We describe how logstash can be used as the shipping layer.  It It will be configured to use rabbitmq redis to ship individual log files to files to a centralized location. Fluentd could be another option.

The shipping phase will interact with another layer called the indexer/search layer. This layer will also enable storing the logs in a format that will help in help in writing search queries. In the current implementation we are proposing Here we describe the use of logstash to recieve the individual log files and elasticsearch to search through search through these. Before logstash outputs the recieved messages to elasticsearch, it will apply a specific grok filter that will split the input messages into key into key value pairs. The key value pair will allow creation of search queries by (key,value). Via the elasticsearch REST api , search queries can be constructed for constructed for required jobid.

Instances of Logstash:

Logstash can aggregate log messages from multiple nodes and multiple log files. In a typical production environment,  cloudstack cloudstack is configured with multiple management server instances for scalability and redundancy. One instance of logstash will be configured to run on each of each of the management server and will ship the log to a AMQP brokerredis. The logstash process is reasonably light in terms of memory consumption and should not impact not impact the management server. 

Instances of elasticsearch and AMQP broker:redis

Elasticsearch runs as a horizontal scale out cluster. The clusters node can be created in two different modes.

...