Introduction

Purpose

This is functional specification for Syslog Enhacements feature of Cloudstack.

References

https://issues.apache.org/jira/browse/CLOUDSTACK-712

Useful Links

Brief Introduction to Syslog

Use cases

These Syslog alert messages can be used by remote Syslog Servers  to find out the problems in Cloudstack managers   to diagnose issues in managed software/systems like CloudStack and can act accordingly. This is similar to alerts that we see on dashboard of CloudstackCloudStack.

1. Admin will set the Ip IP of remote Syslog hosts through Cloudstack CloudStack configuration file, log4j-cloud.xml and those remote Syslog Hosts  will start getting Syslog alert messages then.
2. Admin can  delete the Syslog Hosts by removing them from configuration file

Feature Specifications

In With this feature we are implementing the Syslog enhancements for the plan is to generate Syslog messages for all the supported alerts in CloudStack. In the current scenario we are writing alerts to database and are shown on Cloudstack dashboard. Now in addition to that we will also send the syslog messages in a format which can be easily analyzed by external Syslog messages analyzerThis feature will be in sync with the SNMP feature(proposed) on the alerts/events that will be generated by the CloudStack system. Currently CloudStack generates following alerts/events, these events/alerts are persistent in the CloudStack database and exposed via CS API/UI.

 We will send the Syslog messages for following alerts

...

1. alertType
2. message
3. podId
4. dataCenterId
5. clusterId

Message structure for Syslog messages will be as follows

Date severity_level Management_Server_IP_Address/Name  alertType:: value dataCenterId:: value  podId:: value  clusterId:: value  message:: value

if some keys are not valid(like dataCenterId is 0)  then those will not be set

For ExampleA Sample syslog message would look something like this

Mar  4 10:13:47    WARN    localhost    alertType:: managmentNode   managementNode message:: Management server node 127.0.0.1 is up

Mar  4 10:13:47    WARN   10.1.1.1       alertType:: managmentNode*  * message:: Management network CIDR is not configured originally. Set it default to 10.144.6.0/23

Mar  4 10:13:47    WARN   10.1.1.1       alertType:: domainRouterVmState*  * dataCenterId:: 1  podId:: 1   message:: More than one redundant virtual router is in MASTER state! ...

currently sending  all the alerts to administrator and he will filter alerts according to his needs.

For this feature, we will be using log4j library with Apache License

Severity Level Categorization

Below Alert Types are categorized as CRITICAL

1. host
2. userVmState 
3. domainRouterVmState
4. consoleProxyVmState
5. storageIssueSystemVms
6. ssvmStopped
7. usageServerResult
8. usageServerStatus
9. domainRouterMigrate
10. consoleProxyMigrate
11. routingConnection

while all others are categorized as WARN

Architecture and Design description

This feature will be implemented as plugin. This feature will use the log4j Appender to get the required alerts and will generate the Syslog  messages from that. SyslogAppender of log4j will be used to send the Syslog messages

Classes

We will add following classes

• AlertsSyslogAppender contains mainly two methods
  • append(LoggingEvent) : sends the Syslog message to the configured Syslog Hosts using SyslogAppender class
  • setSyslogAppenders() : sets the Syslog Appenders  to whom for each different Syslog Host

Changes in log4j-cloud.xml

To configure multiple Syslog Hosts one needs to add following appender in log4j-cloud.xml will be added

   <appender name="ALERTSYSLOG">
      <param name="Threshold" value="WARN"/>
      <param name="SyslogHosts" value=""/>
      <param name="Facility" value="LOCAL6"/>
      <layout>
         <param name="ConversionPattern" value=""/>
      </layout>
   </appender>To appender>

To specify multiple  Syslog Hosts one , admin  has to modify in above fragment as follows with each Syslog Host  separated by ,

   <appender name="ALERTSYSLOG">
      <param name="Threshold" value="WARN"/>
      <param name="SyslogHosts" value="10.1.1.1,10.1.1.2"/>
      <param name="Facility" value="LOCAL6"/>       <layout>
         <param name="ConversionPattern" value=""/>
      </layout>
   </appender>

Following loggers will also be added      

   <logger name="com.cloud.alert.AlertManagerImpl" additivity="false">
      <level value="WARN"/>
      <appender-ref ref="SYSLOG"/>
      <appender-ref ref="CONSOLE"/>
      <appender-ref ref="FILE"/>
      <appender-ref ref="ALERTSYSLOG"/>
   </logger>
   <logger name="com.cloud.usage.UsageAlertManagerImpl" additivity="false">
      <level value="WARN"/>
      <appender-ref ref="SYSLOG"/>
      <appender-ref ref="CONSOLE"/>
      <appender-ref ref="FILE"/>
      <appender-ref ref="ALERTSYSLOG"/>
   </logger>

UI flow

There will be no UI/API in this feature