...
Root Admin users can query this service to quickly identify the sequence of steps related to a particular Job.
QA can use this service to link the log messages related to automated test failures.
...
The system will comprise of a log shipping layer. This layer will be responsible for collecting logs from each of the management server shipping them to a centralized place. We describe how logstash can be used as the shipping layer. It will be configured to use redis to ship individual log files to a centralized location. Fluentd could be another option.
...
=====================================================================================================
API Command :
A new API command ExtractLogByJobIdCmd will be introduced. This will be implemented as a synchronous command.
The following two scripts can automate steps 1 to 5 for the above process.
This script takes all the paramters in a setup config file. The only line param this requires is the config file. Default setup.conf is adequately commented
to describe the required paramsThe manager class will implement the actual functionality of querying elastic search for log messages that match the specified filters. For doing this the Elasticsearch REST api queries will be used. Post method will be used with elasticsearch DSL to specify the required query. DSL is quite flexible and in future if support is required to filter by time stamp and other values DSL would help achieve that with ease.
DSL query for searching logs by jobid
...