Page History

Bug Reference

https://issues.apache.org/jira/browse/CLOUDSTACK-3471

Branch

master

Introduction

As of cloudstack 4.3 there is no api that can aggregate log messages by the job id. Since logging is a typically a service that can be and is usually available outside the IaaS core, it was felt that instead of integrating this as an api within cloudstack, it would be better to provide a generic logsearch service that could be invoked by cloudstack to retrieve log messages. We describe the mechanism on how to achieve this in what follows.

...

This is typically useful to root admin users.

References

Document History

Glossary

Feature Specifications

Use cases

1. Root Admin users can query this service to quickly identify the sequence of steps related to a particular Job.

2. QA can use this service to link the log messages related to automated test failures.

Layout Description

The system will comprise of a log shipping layer. This layer will be responsible for collecting logs from each of the management server shipping them to a centralized place. We describe how logstash can be used as the shipping layer. It will be configured to use redis to ship individual log files to a centralized location. Fluentd could be another option.

...

=====================================================================================================

Steps in setting up the service:

1. Deploy a vm intance from any existing linux based user template
2. setup logstash on this guest instance
   1. There is a an apt repository and a yum repository from which logstash can be deployed.
3. setup elasticsearch on this guest instance
   1. There is a an apt repository and a yum repository from which logstash can be deployed.
4. Setup redis on this guest instance
   1. For yum : This is available from the REMI repository
   2. For apt : This is available from the
5. Create a template from this virtual machine
6. Subsequently this template can be used to spawn elasticsearch master and slave nodes.

Automation:

The following two scripts can automate steps 1 to 5 for the above process.

...

This script takes all the paramters in a setup config file. The only line param this requires is the config file. Default setup.conf is adequately commented
to describe the required params.

Default setup.conf:

#########################################################################################################
#This is the base template which will be used to provision a guest with logstash,elasticsearch and redis
templateid=

...

{
    "query": {
        "query_string": {
            "query": "\<jobid\>",
   "fields" : "jobid"
        }
    }
}

UI flow

None

IP Clearance

• Logstash which is an opensource log management tool and is covered under Apache 2.0 license
• Elasticsearch which is a search and analytics tool and is again covered under Apache 2.0 license. 

Appendix

Appendix A:Appendix B: 

Labels: