Introduction

Purpose

This is functional specification for Syslog Enhacements feature of Cloudstack.

References

https://issues.apache.org/jira/browse/CLOUDSTACK-712

Brief Introduction to Syslog

Use cases

Syslog messages can be used by remote Syslog managers   to diagnose issues in managed software/systems like CloudStack and can act accordingly. This is similar to alerts that we see on dashboard of CloudStack.

  1. Admin will set the IP of remote Syslog hosts through CloudStack configuration file, log4j-cloud.xml and those remote Syslog Hosts  will start getting Syslog alert messages then.
  2. Admin can  delete the Syslog Hosts by removing them from configuration file

Feature Specifications

With this feature the plan is to generate Syslog messages for all the supported alerts in CloudStack. This feature will be in sync with the SNMP feature(proposed) on the alerts/events that will be generated by the CloudStack system. Currently CloudStack generates following alerts/events, these events/alerts are persistent in the CloudStack database and exposed via CS API/UI.

 We will send the Syslog messages for following alerts

  1. availableMemory : Available Memory below configured threshold
  2. availableCpu :  Unallocated CPU below configured threshold
  3. availableStorage :  Available Storage below configured threshold
  4. remainingStorageAllocated : Remaining unallocated Storage is below configured threshold
  5. unallocatedVirtualNetworkpublicIp : Number of unallocated virtual network public IPs is below configured threshold
  6. unallocatedPrivateIp : Number of unallocated private IPs is below configured threshold
  7. availableSecondaryStorage :  Available Secondary Storage in availability zone is below configured threshold
  8. host : host related alerts like host disconnected etc
  9. userVmState : User VM stopped unexpectedly
  10. domainRouterVmState : Domain Router VM stopped unexpectedly
  11. consoleProxyVmState : Console Proxy VM stopped unexpectedly
  12. routingConnection : lost connection to default route (to the gateway)
  13. storageIssueSystemVms : storage issue in system VMs
  14. usageServerStatus : No usage server process running
  15. managmentNode : Management network CIDR is not configured originally
  16. domainRouterMigrate : Domain Router VM Migration was unsuccessful
  17. consoleProxyMigrate : Console Proxy VM Migration was unsuccessful
  18. userVmMigrate  : User  VM Migration was unsuccessful
  19. unallocatedVlan : Number of unallocated VLANs is below configured threshold in availability zone
  20. ssvmStopped : SSVM stopped unexpectedly
  21. usageServerResult :  Usage job failed
  22. storageDelete : Failed to Delete storage pool
  23. updateResourceCount : Failed to update the resource count
  24. usageSanityResult : Usage Sanity Check failed
  25. unallocatedDirectAttachedPublicIp : Number of unallocated shared network IPs is low in availability zone
  26. unallocatedLocalStorage : Remaining unallocated Local Storage is below configured threshold
  27. resourceLimitExceeded : Resource limit exceeds the limit

Each message will contain the following keys and corresponding value

  1. alertType
  2. message
  3. podId
  4. dataCenterId
  5. clusterId

Message structure for Syslog messages will be as follows

Date severity_level Management_Server_IP_Address/Name  alertType:: value dataCenterId:: value  podId:: value  clusterId:: value  message:: value

if some keys are not valid(like dataCenterId is 0)  then those will not be set

A Sample syslog message would look something like this

Mar  4 10:13:47    WARN    localhost    alertType:: managementNode message:: Management server node 127.0.0.1 is up

For this feature, we will be using log4j library with Apache License

Severity Level Categorization

Below Alert Types are categorized as CRITICAL

  1. host
  2. userVmState 
  3. domainRouterVmState
  4. consoleProxyVmState
  5. storageIssueSystemVms
  6. ssvmStopped
  7. usageServerResult
  8. usageServerStatus
  9. domainRouterMigrate
  10. consoleProxyMigrate
  11. routingConnection

while all others are categorized as WARN

Architecture and Design description

This feature will be implemented as plugin. This feature will use the log4j Appender to get the required alerts and will generate the Syslog  messages from that. SyslogAppender of log4j will be used to send the Syslog messages

Classes

We will add following classes

  • AlertsSyslogAppender contains mainly two methods
    • append(LoggingEvent) : sends the Syslog message to the configured Syslog Hosts using SyslogAppender class
    • setSyslogAppenders() : sets the Syslog Appenders  to whom for each different Syslog Host

Changes in log4j-cloud.xml

To configure multiple Syslog Hosts following appender in log4j-cloud.xml will be added

   <appender name="ALERTSYSLOG">
      <param name="Threshold" value="WARN"/>
      <param name="SyslogHosts" value=""/>
      <param name="Facility" value="LOCAL6"/>
      <layout>
         <param name="ConversionPattern" value=""/>
      </layout>
   </appender>

To specify multiple  Syslog Hosts, admin  has to modify in above fragment as follows with each Syslog Host  separated by ,

   <appender name="ALERTSYSLOG">
      <param name="Threshold" value="WARN"/>
      <param name="SyslogHosts" value="10.1.1.1,10.1.1.2"/>
      <param name="Facility" value="LOCAL6"/>       <layout>
         <param name="ConversionPattern" value=""/>
      </layout>
   </appender>

Following loggers will also be added      

   <logger name="com.cloud.alert" additivity="false">
      <level value="WARN"/>
      <appender-ref ref="SYSLOG"/>
      <appender-ref ref="CONSOLE"/>
      <appender-ref ref="FILE"/>
      <appender-ref ref="ALERTSYSLOG"/>
   </logger>
   <logger name="com.cloud.usage" additivity="false">
      <level value="WARN"/>
      <appender-ref ref="SYSLOG"/>
      <appender-ref ref="CONSOLE"/>
      <appender-ref ref="FILE"/>
      <appender-ref ref="ALERTSYSLOG"/>
   </logger>

UI flow

There will be no UI/API in this feature

  • No labels