You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 10 Next »

Introduction

Purpose

This is functional specification for Syslog Enhacements feature of Cloudstack.

References

https://issues.apache.org/jira/browse/CLOUDSTACK-712

Brief Introduction to Syslog

Use cases

Syslog messages can be used by remote Syslog managers   to diagnose issues in managed software/systems like Cloudstack and can act accordingly. This is similar to alerts that we see on dashboard of Cloudstack.

  1. Admin will set the Ip of remote Syslog hosts through Cloudstack configuration file, log4j-cloud.xml and those remote Syslog Hosts  will start getting Syslog alert messages then.
  2. Admin can  delete the Syslog Hosts by removing them from configuration file

Feature Specifications

In this feature we are implementing the Syslog enhancements for alerts in CloudStack. In the current scenario we are writing alerts to database and are shown on Cloudstack dashboard. Now in addition to that we will also send the syslog messages in a format which can be easily analyzed by external Syslog messages analyzer.

 We will send the Syslog messages for following alerts

  1. availableMemory : Available Memory below configured threshold
  2. availableCpu :  Unallocated CPU below configured threshold
  3. availableStorage :  Available Storage below configured threshold
  4. remainingStorageAllocated : Remaining unallocated Storage is below configured threshold
  5. unallocatedVirtualNetworkpublicIp : Number of unallocated virtual network public IPs is below configured threshold
  6. unallocatedPrivateIp : Number of unallocated private IPs is below configured threshold
  7. availableSecondaryStorage :  Available Secondary Storage in availability zone is below configured threshold
  8. host : host related alerts like host disconnected etc
  9. userVmState : User VM stopped unexpectedly
  10. domainRouterVmState : Domain Router VM stopped unexpectedly
  11. consoleProxyVmState : Console Proxy VM stopped unexpectedly
  12. routingConnection : lost connection to default route (to the gateway)
  13. storageIssueSystemVms : storage issue in system VMs
  14. usageServerStatus : No usage server process running
  15. managmentNode : Management network CIDR is not configured originally
  16. domainRouterMigrate : Domain Router VM Migration was unsuccessful
  17. consoleProxyMigrate : Console Proxy VM Migration was unsuccessful
  18. userVmMigrate  : User  VM Migration was unsuccessful
  19. unallocatedVlan : Number of unallocated VLANs is below configured threshold in availability zone
  20. ssvmStopped : SSVM stopped unexpectedly
  21. usageServerResult :  Usage job failed
  22. storageDelete : Failed to Delete storage pool
  23. updateResourceCount : Failed to update the resource count
  24. usageSanityResult : Usage Sanity Check failed
  25. unallocatedDirectAttachedPublicIp : Number of unallocated shared network IPs is low in availability zone
  26. unallocatedLocalStorage : Remaining unallocated Local Storage is below configured threshold
  27. resourceLimitExceeded : Resource limit exceeds the limit

Each message will contain the following keys and corresponding value

  1. alertType
  2. message
  3. podId
  4. dataCenterId
  5. clusterId

Message structure for Syslog messages will be as follows

Date severity_level Management_Server_IP_Address/Name  alertType:: value dataCenterId:: value  podId:: value  clusterId:: value  message:: value

if some keys are not valid(like dataCenterId is 0)  then those will not be set

For Example

Mar  4 10:13:47    WARN    localhost    alertType:: managementNode   message:: Management server node 127.0.0.1 is up

Mar  4 10:13:47    WARN   10.1.1.1       alertType:: managementNode   message:: Management network CIDR is not configured originally. Set it default to 10.144.6.0/23

Mar  4 10:13:47    WARN   10.1.1.1       alertType:: domainRouterVmState   dataCenterId:: 1  podId:: 1   message:: More than one redundant virtual router is in MASTER state! ...

currently sending  all the alerts to administrator and he will filter alerts according to his needs.

For this feature, we will be using log4j library with Apache License

Severity Level Categorization

Below Alert Types are categorized as CRITICAL

  1. host
  2. userVmState 
  3. domainRouterVmState
  4. consoleProxyVmState
  5. storageIssueSystemVms
  6. ssvmStopped

while all others are categorized as WARN

Architecture and Design description

This feature will be implemented as plugin. This feature will use the log4j Appender to get the required alerts and will generate the Syslog  messages from that. SyslogAppender of log4j will be used to send the Syslog messages

Classes

We will add following classes

  • AlertsSyslogAppender contains mainly two methods
    • append(LoggingEvent) : sends the Syslog message to the configured Syslog Hosts using SyslogAppender class
    • setSyslogAppenders() : sets the Syslog Appenders  to whom for each different Syslog Host

Changes in log4j-cloud.xml

To configure multiple Syslog Hosts following appender in log4j-cloud.xml will be added

   <appender name="ALERTSYSLOG">
      <param name="Threshold" value="WARN"/>
      <param name="SyslogHosts" value=""/>
      <param name="Facility" value="LOCAL6"/>
      <layout>
         <param name="ConversionPattern" value=""/>
      </layout>
   </appender>

To specify multiple  Syslog Hosts, admin  has to modify in above fragment as follows with each Syslog Host  separated by ,

   <appender name="ALERTSYSLOG">
      <param name="Threshold" value="WARN"/>
      <param name="SyslogHosts" value="10.1.1.1,10.1.1.2"/>
      <param name="Facility" value="LOCAL6"/>       <layout>
         <param name="ConversionPattern" value=""/>
      </layout>
   </appender>

Following loggers will also be added      

   <logger name="com.cloud.alert.AlertManagerImpl" additivity="false">
      <level value="WARN"/>
      <appender-ref ref="SYSLOG"/>
      <appender-ref ref="CONSOLE"/>
      <appender-ref ref="FILE"/>
      <appender-ref ref="ALERTSYSLOG"/>
   </logger>
   <logger name="com.cloud.usage.UsageAlertManagerImpl" additivity="false">
      <level value="WARN"/>
      <appender-ref ref="SYSLOG"/>
      <appender-ref ref="CONSOLE"/>
      <appender-ref ref="FILE"/>
      <appender-ref ref="ALERTSYSLOG"/>
   </logger>

  • No labels