Page History

Logging Technical Specification

1 Introduction

This document describes the modernization of GemFire logging to use Log4J 2. This will introduce usage of categories and markers to the product which will have a major impact on improving debugging by engineering as well as customers. It will also greatly improve our integration in other environments such as Tomcat/TC-Server.

GemFire releases 1 through 3 were a much smaller product with a much smaller localized team. LogWriter and its limitations were sufficient for that team and the much smaller original feature set. Enabling FINE wouldn’t fill your disk and it facilitated debugging sufficiently. GemFire 1 was also released at a time before the current logging libraries were created. We now have multiple teams separated geographically, and the size of the product has grown exponentially. GemFire has simply outgrown the usefulness of the old LogWriter logging. In order to debug and grow the logging to be useful again, we need to adopt one of the new logging libraries with categories, mapped diagnostic contexts, markers and filters.

Driving requirements: a) improve debugging by introducing full-qualified class name categories, mapped diagnostic contexts, markers, filters, flow tracing, static loggers and other features missing in GemFire logging, b) improve integration with customer applications and deployment environments such as Tomcat/tcServer, c) provide industry standard logging API as requested by many customers and field engineers, d) improve code maintenance by replacing deficient proprietary logging with free, open-source, modern logging library, e) select the best performing library that meets the requirements.

2 Bugs and Feature Requests 

This section lists the Bugs and Feature Requests that also need to be addressed as part of this overall effort:

• 2.1 #47722 G0 Configurable option of appending to log file or creating new file
• 2.2 #50483 F0 log file size does not account for user/application logging
• 2.3 #51111 F0 Web application (Tomcat) logging goes to gemfire log
• 2.4 #47910 R2 <Regression> ConcurrentModificationException in createLogWriter during locator startup.
• 2.5 #51207 F2 Improve logging configuration for libraries
• 2.6 #40414 R3 i18n changes (toLocalizedString) causes ports, ids to be formatted with commas
• 2.7 #43396 R3 Remove dead messages from LocalizedStrings and fix any obvious typos
• 2.8 #51197 C2 A log file name with no '.log' extension causes the gfxd connect to fail after the first time
• 2.9 #45796 F3 Feature Request:Adding a Functionality to Make Logging Custom Filter Available in Gemfire

Introducing Log4J 2 as our underlying logging library satisfies 2.1. #47722 and 2.5. #51207. In addition it makes 2.4. #47910 go away.

Removing OSProcess.redirectCOutput makes #50483 irrelevant. See 67.10.1.

TODO: verify that Log4J 2 fulfills #48796 as well. Log4J 2 certainly provides robust filtering but I’m not sure if you can add does it support java.util.logging.Filters specifically into Log4J 2.

also?

3 Related Documents 

• Logging Thoughts
• Logging Backlog (do we have a better tool than a spreadsheet?)
  

4 Terminology 

• FQCN -- Fully-qualified class name which is used as the name of the logger that is declared at the top of the class or the class’s root ancestor. Example:

...

6 Overview

Summary of the changes:

6.1 GemFire product code will be updated to use a Logger defined in every class or each class’s root ancestor using the fully-qualified class name as the name of its Logger. This is the industry standard in Java logging and it provides categorical control over logging which dramatically improves debugging for engineering, support and customers.

6.2 Custom log levels introduce too great of a performance impact, so FINEST and FINER will map to TRACE, FINE will map to DEBUG statements previously logged at CONFIG will be logged at INFO.

6.3 Log statements should be updated to use Log4J 2’s parameterization:

...

if (log.isDebugEnabled()) {   
  log.debug("Hi, " + string1 + " " + string2);
}

...

Becomes:

log.debug("Hi, {} {}", string1, string2);

...

See 7.6. for more details.

6.4 Markers will be used to further organize logging by component or service layer in order to facilitate filtering and debugging beyond what is already provided by fully-qualified class name categories. Markers can extend other Markers, but a log statement cannot specify two unrelated Markers. I propose adding an interface to com.gemstone.gemfire.logging which defines all of our product Markers with extensive javadocs so that users can a) be aware of the Markers so they know what they can filter on, b) log with our Markers if they so desire. TODO: specify a possible set of Markers in section 6

6.5 The old LogWriter API and logging properties in gemfire.properties will be deprecated. Current product defaults have these disabled except to stdout so those defaults will remain unchanged. If someone is determined to use those old properties then they can can still enable logging of LogWriter to file. The InternalDistributedSystem will register a LogWriterAppender which is configured with the appropriate “log-” properties. Because this is a Log4J 2 appender, the filtering by categories, markers, etc will allow the user to control its output more thoroughly.

6.6 By default the new Log4J 2 logging will be enabled for logging to stdout. The standard way of configuring Log4J 2 is with a log4j2.xml and we won’t change that. GemFire will include a copy of its default log4j2.xml configuration of Log4J 2 in GEMFIRE/defaultConfigs with a PatternLayout that makes the log message format the same as what LogWriter produces. The com.gemstone.gemfire.internal.logging.log4j package will include a copy of that same log4j2.xml as a resource to use if no other configuration is available. If user has already configured Log4J 2 then the user’s configuration will be used. If it has not been configured then the search order of precedence will be a) specified by user with the standard Log4J 2 system property, b) current working directory of the GemFire process, c) GEMFIRE/defaultConfigs (KIRK: delete this from search path but leave it there as an example), d) com.gemstone.gemfire.logging.internal.log4j.log4j2.xml resource inside the gemfire.jar.

6.7 GemFire will have an absolute required dependency on the Log4J 2 API jar which is currently log4j-api-2.0.2.jar. The default GemFire log4j2.xml includes configuration of appenders and pattern layout that duplicates the old syntax of GemFire LogWriter output, Alerting, Security, file rolling, etc. These details require that we use and implement classes, including custom layouts and appenders, that make use of the Log4J 2 Core jar which is currently log4j-core-2.0.2.jar. We hope to have an optional (though highly recommended) dependency on the Core jar. Even if we are able to have an optional dependency on this jar, it will be necessary in order to get any logging output that resembles the legacy output format. The reason is that the product code will be changed to use Log4J 2 Loggers directly rather than some facade that can optionally adapt to either old LogWriter or new Log4J 2 with the LogWriterAppender. We did some performance testing and found that such a facade introduces a drop in performance.

7 Architectural Description

...