Description

Java application VMs allocate objects on the heap. The heap includes both live and dead objects. Quantity of heap space is configured using the -Xms and -Xmx VM arguments. When the heap becomes close to full, GCs occur which can cause application pauses.

Determination

Running out of heap space is a heap memory issue that will manifest itself in the Geode log file as an OutOfMemoryError with the message Java heap space thrown either by a Geode thread or an application thread. The error must contain the Java heap space message and not the unable to create new native thread message (see Troubleshooting Native Memory for details on that issue). An example is shown below.

[severe 2009/02/23 19:38:48.293 UTC <Gateway Event Processor> tid=0xbd] Uncaught exception in thread <Gateway Processor>
 java.lang.OutOfMemoryError: Java heap space

 

vsd

Another way to determine whether there is a heap memory issue is to use vsd to display free, used and maximum heap memory values contained in a given Geode statistics archive.

vmHeapMemoryStats

The chart below shows VMMemoryUsageStats-vmHeapMemoryStats maxMemory and usedMemory values. The VMMemoryUsageStats values show the overall heap memory usage.

VMMemoryUsageStats

CMS Old Gen-Heap memory

The chart below shows VMMemoryPoolStats-CMSOldGen-Heapmemory currentMaxMemory and currentUsedMemory values. The VMMemoryPoolStats values show the heap memory usage for specific memory pools (i.e. CMS).

VMMemoryPoolStats

VMStats

The chart below shows the VMStats freeMemory values. The VMStats freeMemory values show the JVM's free memory. 

VMStats

gfsh

The gfsh show metrics command can be used to show the current (currentHeapSize) and maximum (maximumHeapSize) heap sizes of a member. An example is:

show metrics --member=server1 --categories=member
 
Member Metrics

Category |     Metric      | Value
-------- | --------------- | -----------------
member   | upTime          | 1638
         | cpuUsage        | 89.60260009765625
         | currentHeapSize | 6800
         | maximumHeapSize | 81250

Action

There are several ways to eliminate heap issues, including:

  • Increase the maximum amount of heap available to the VM using the -Xmx VM argument
  • Configure one or more regions to overflow so that their values are not stored on the heap
  • Scale the distributed system horizontally by adding one or more additional members
  • No labels