Revision History
First draft | 04/19/2013 | Kirk Lund | |
Second draft | 05/02/2013 | Kirk Lund | |
Third draft | 06/07/2013 | Kirk Lund | Defined OutOfMemoryError for off-heap, removed ALTER TABLE, added more stats, expanded ResourceManager and Eviction |
Fourth draft | 06/11/2013 | Kirk Lund | Add changes to ResourceAdvisor |
Fifth draft | 10/03/2013 | Kirk Lund | Updated statistics and information about MBean changes |
Sixth draft | 02/04/2015 | Kirk Lund | Mostly minor edits to better clarify certain points |
7th draft | 05/11/2015 | Kirk Lund | Update section 5.6 with range index throws UnsupportedOperationException; add section 5.7 OperationContext; add section 5.8 Result of Region.put, Region.destroy, Region.remove, and Region.Entry.setValue; add section 5.9 PartitionAttributes localMaxMemory |
1 Introduction
2 Related Documents
External sources:
3 Terminology
- off-heap memory – memory that is not on the standard Java heap and is not managed by Java and its garbage collector
4 Architectural Description
The goals and general approach to using off-heap is described in MemScale (high-level charter/goals).
Experimentation reveals that usage of off-heap memory provides a potentially significant performance boost by minimizing CPU usage of the Java garbage collector. The extra processing required to serialize/deserialize and store/fetch the data through sun.misc.Unsafe is offset by having less Java heap memory for the garbage collector to manage as well as not consuming the CPU cycles normally used up by the collector.
4.1 Configuration
- off-heap-memory-size – new GemFire property to specify size of memory in kilobytes, megabytes or gigabytes to allocate for off-heap storage
- lock-memory – new GemFire property that allows both heap and off-heap memory to be locked in memory
- off-heap – new Region attribute to specify usage of off-heap memory for Region entries
4.1.1 off-heap-memory-size GemFire property
* <b>Off-Heap Memory</b>
*
* <dl>
* <a name="off-heap-memory-size"><dt>off-heap-memory-size</dt></a>
* <dd><U>Description</U>: The total size of off-heap memory specified as
* off-heap-memory-size=<n>[g|m]. <n> is the size. [g|m] indicates
* whether the size should be interpreted as gigabytes or megabytes.
* </dd>
* <dd><U>Default</U>: <code>""</code></dd>
* <dd><U>Since</U>: 9.0</dd>
* </dl>
Examples of valid values:
- off-heap-memory-size=4096m
- off-heap-memory-size=120g
4.1.2 lock-memory GemFire property
* <dl>
* <a name="lock-memory"><dt>lock-memory</dt></a>
* <dd><U>Description</U>: Set to true to lock GemFire heap and off-heap memory pages into RAM.
* This prevents the operating system from swapping the pages out to disk, which can cause severe performance degradation. * </dd> * <dd><U>Default</U>: false</dd> * <dd><U>Since</U>: 9.0</dd> * </dl>
4.1.3 off-heap Region attribute
4.1.2.1 Declarative configuration of off-heap
<!ATTLIST region-attributes ... off-heap (false | true) #IMPLIED >
4.1.2.2 Programmatic configuration of off-heap
/** * Enables this region's usage of off-heap memory if true. * @since 9.0 * @param offHeap boolean flag to enable off-heap memory */ public void setOffHeap(boolean offHeap) {
/** * Returns whether or not this region uses off-heap memory. * @return True if a usage of off-heap memory is enabled; * false if usage of off-heap memory is disabled (default). * @since 9.0 */ public boolean getOffHeap();
4.2 Calculation of Slab Size
Our implementation organizes the off-heap memory into slabs. Rather than expose another configuration detail to the user, we will calculate the best slab size possible.
We believe that Integer.MAX_VALUE (about 2 GB) should be an acceptable maximum slab size. This determines that 2 GB is the maximum sized object or data allowed.
For simplicity we are making the slabs all of equal size. Therefore the off-heap memory must be divisible by the calculated slab size with negligible waste.
If the total off-heap memory is 2 GB or less, we will have one slab. If the off-heap memory is greater than 2 GB then there will be 2 or more slabs.
public static long calcSlabSize(long offHeapMemorySize) { final String offHeapSlabConfig = System.getProperty("gemfire.OFF_HEAP_SLAB_SIZE"); long slabSize = 0; if (offHeapSlabConfig != null && !offHeapSlabConfig.equals("")) { slabSize = parseLongWithUnits(offHeapSlabConfig, 100*1024*1024, 1024*1024); } else { // calculate slabSize if (offHeapMemorySize < (2*GIGABYTE)) { // 2 GB or under has one slab slabSize = offHeapMemorySize; } else if (offHeapMemorySize == (2*GIGABYTE)) { slabSize = Integer.MAX_VALUE; } else { // over 2 GB has more than one slab final long modulus = offHeapMemorySize % (2*GIGABYTE); if (modulus == 0) { // 2 GB divides cleanly slabSize = Integer.MAX_VALUE; } else { final long numSlabs = (offHeapMemorySize + modulus) / (2*GIGABYTE); slabSize = offHeapMemorySize / numSlabs; } } } return slabSize; }
4.3 Off-Heap Memory statistics
name | description |
---|---|
compactions | The total number of times off-heap memory has been compacted. |
compactionTime | The total time spent compacting off-heap memory. |
fragmentation | The percentage of off-heap memory fragmentation. Updated every time a compaction is performed. |
fragments | The number of fragments of free off-heap memory. Updated every time a compaction is done. |
freeMemory | The amount of off-heap memory, in bytes, that is not being used. |
largestFragment | The largest fragment of memory found by the last compaction of off heap memory. Updated every time a compaction is done. |
maxMemory | The maximum amount of off-heap memory, in bytes. This is the amount of memory allocated at startup and does not change. |
objects | The number of objects stored in off-heap memory. |
reads | The total number of reads of off-heap memory. |
usedMemory | The amount of off-heap memory, in bytes, that is being used to store data. |
4.4 M&M Support
4.4.1 JMX Changes
MemberMXBean has new attributes which expose several of the off-heap memory statistics for the member.
- MemberMXBean
- getOffHeapCompactionTime - provides the value of the compactionTime statistic
- getOffHeapFragmentation - provides the value of the fragmentation statistic
- getOffHeapFreeMemory - provides the value of the freeMemory statistic
- getOffHeapObjects - provides the value of the objects statistic
- getOffHeapUsedMemory - provides the value of the usedMemory statistic
- getOffHeapMaxMemory - provides the value computed by freeMemory + usedMemory
If the compactionTime is of non-zero value then the member has had to compact the off-heap memory. This may indicate that the member is close to running out of off-heap memory. It also indicates how much time the GemFire XD process is having to spend compacting. Does this help with capacity planning or analyzing usage patterns? Lots of variability in row sizes might possibly result in more compacting than row sizes that are more uniform.
FreeMemory + UsedMemory is the same as MaxMemory so we did not expose MaxMemory as an attribute (it would never change value anyway). Note: the MemberMXBean operation listGemFireProperties would include off-heap-memory-size which is also the same as MaxMemory.
Objects show the number of GemFire objects stored in the Region.
In theory, Fragmentation would be a valuable attribute to monitor. An excessively high Fragmentation percentage would indicate that the off-heap memory is sufficiently fragmented to potentially make it difficult to fill up or utilize. This means the member might become critical (see docs about ResourceManager) and throw LowMemoryException. It's also possible that a member may throw OutOfOffHeapMemoryException, which will cause the cache to close in order to maintain consistency across the cluster. The latter is last resort and each member will attempt to use the off-heap eviction and critical thresholds on the ResourceManager to prevent data inconsistency or member loss. This may deserve further discussion (it's complicated).
RegionMXBean listRegionAttributes includes a new boolean attribute that indicates if the region is configured to use off-heap memory or regular Java heap memory.
- RegionMXBean
- one of the attributes returned by the listRegionAttributes operation is offHeap which will have a value of true if that region is configured to use off-heap memory (GFXD: was enableOffHeapMemory)
4.4.2 Pulse Support
Changes to Pulse for GemFireXD Off-Heap will need to be ported with minor changes if any. If possible, we should expose all the off-heap attributes on MemberMXBean including the new getOffHeapMaxMemory.
Each region using off-heap should also be flagged as an off heap region. The attribute was "enable-off-heap-memory" in GFXD, but will be "off-heap" in GemFire.
4.4.3 GFSH Command Changes
- alter disk-store: new option "--off-heap" for setting off-heap for each region in the disk-store
- create region: new option "--off-heap" for setting off-heap
- describe member: now displays the off-heap size
- describe offline-disk-store: now shows if a region is off-heap
- describe region: now displays the off-heap region attribute
- show metrics: Now has an offheap category. The offheap metrics are: maxMemory, freeMemory, usedMemory, objects, fragmentation, and compactionTime
- start server: added --lock-memory, --off-heap-memory-size, --critical-off-heap-percentage, and --eviction-off-heap-percentage
4.5 Design Limits and Constraints
Discuss minimum and maximum configuration and usage. Most of the limits are due to the max size of primitives being used, such as integer for number of slabs or long for memory address pointers.
4.5.1 Maximum Object Size
The maximum off-heap data size is 2 GB, but some of this data includes implementation-specific meta-data which lowers the maximum serialized user data (bytes) to less than 2 GB.
4.5.2 All Members Must Configure Off-Heap Memory Enabled Uniformly
We will not be allowing two members to host data for the same Region with a different value for off-heap enabled. We stepped through some scenarios and the ResourceManager and LRU settings just didn't work properly unless the members were required to be configured the same way. TODO: this constraint may change for rolling a region to off-heap
4.6 Region Entries
Equality checks are done without deserializing the value stored on off-heap memory. If the value is serialized using pdx then PdxInstance equals is used for comparison. Otherwise the raw serialized bytes are compared. This means that custom equals() implementations will not be called when checking off-heap values for equality. Value equality checks are done by Region.replace(K,V,V) and Region.remove(Object, Object).
4.7 Off-Heap Object Details
Off-heap memory is going to perform best for uniform value sizes as this minimizes the impact of fragmentation. We are using a compaction strategy which does not move live data. So usage patterns involving mass creates followed by mass destroys (if destroying at all) will also minimize effects of fragmentation.
The worst usage pattern will be those that create and then perform updates with lots of different size values without every destroying or clearing. This may result in severe fragmentation that our compaction will struggle with.
Compaction involves moving address spaces to a free list from which future creates will pull from. There are 16384 free lists, one for each value size in powers of 8 up to the power of 16KB (8^16384) which allows for a max size up to 131,072. Beyond this the implementation uses a ConcurrentSkipList for the free list of larger chunks. Contiguous entries in a free list can be combined. The compaction of free spaces involves removing them from specific free lists so they can be used for any size.
The number of free lists for values smaller than what's ultimately stored in the ConcurrentSkipList can be changed by setting the System property gemfire.OFF_HEAP_FREE_LIST_COUNT. We do not intend to expose this to the customer and will only use this to enable testing and profiling for determining the best number of free lists.
5.1 Callback EntryEvents
Callback Events (EntryEvents) are not fully usable beyond the scope of the callback on an off-heap region. Attempting to invoke methods which acquire values may throw an exception indicating that the value is no longer available if called after the callback returns. The exception is: IllegalStateException("Attempt to access off heap value after the EntryEvent was released.").
If the user needed to save a reference to the instance of EntryEvent into a variable for use after the CacheListener callback returns, then they will instead need to create an on-heap copy of the value to ensure that the value remains available to them.
For example, invoking com.gemstone.gemfire.cache.EntryEvent#getNewValue() is only safe while the thread is executing within the com.gemstone.gemfire.cache.CacheListener#afterUpdate(EntryEvent) method or other callback method. The EntryEvent methods getOldValue(), getSerializedOldValue(), and getSerializedNewValue() are similarly affected.
5.2 ResourceManager
Existing ResourceManager thresholds would still apply because off-heap values have a heap foot-print which could trigger the existing thresholds.
We will also introduce new ResourceManager support for off-heap memory thresholds (eviction and critical) which will trigger thresholds specific to off-heap memory. The eviction threshold will support all eviction actions. The critical threshold will result in LowMemoryExceptions thrown when attempting write operations on an off-heap table/region.
We need two sets of thresholds because a GemFire node may have a mix of regions that use off-heap entries and regions that use heap entries.
The ResourceAdvisor, profiles and message will require modification to include off-heap memory in addition to Java heap memory. Low memory messages will be sent separately for heap and off-heap memory status changes. Regions using off-heap will then throw LowMemoryException when either heap or off-heap is critical. Regions using heap will throw LowMemoryException only when heap is critical.
5.3 Compression
Compression can be used with an off-heap region.
5.4 Delta Updates
Every time a delta is applied it will need to: deserialize the off-heap value, apply the delta, and re-serialize the value to off-heap. So off-heap with delta has to do a lot more work than heap with delta (since heap keeps the object form in memory letting it apply the delta without doing a deserialize and serialize).
5.5 Querying/OQL
Use with querying/OQL will probably cause frequent bulk copying to the heap.
5.6 Indexes
Off-heap regions with indexes are supported. In the first implementation the index data for an off-heap region will be on the heap so heap usage will be less but can still be high depending on the type and number of indexes.
- hash index - supported
- primary key index - supported
- functional index
- range index - problematic since it copies the value. Since it would copy all the off-heap memory to heap we decided to not support it. If a customer tries to create one on an off-heap region it will throw an UnsupportedOperationException. This is what the product already did for these indexes on an overflow region. At some point in the future we may add support for these indexes to off-heap.
- compact range index - supported
5.7 OperationContext
5.8 Result of Region.put, Region.destroy, Region.remove, and Region.Entry.setValue
5.9 PartitionAttributes localMaxMemory
6 Performance Requirements
General requirements and purpose for the off-heap memory feature:
- no worse than using Java heap for JVMs hosting certain volume of data
- allow certain features to have worse performance: deltas, querying/OQL
- primary purpose is to increase data density while minimizing GC overhead
- allow the customer to use more than 70% memory and sustained CPU maxing without risking GC pauses
6.2 Scalability Goals
- manage 50 GB of customer data in a single member - initial experiments show we can surpass this
- manage 10 TB of customer data in one GemFire cluster - initial experiments attained 1 TB with 8 members
7 Testing
- emphasis on primary usage pattern with cycles of many creates followed by clear (or many destroys)
- ensure that all GemFire features work even if performance is compromised for certain features such as deltas
- ensure capability of scaling to 50 GB of user data in a single member and 10 TB in one cluster
8 Documentation / Customer Recommendations
In general we recommend against combining the use of off-heap memory with the following due to performance and memory overhead implications:
- do not use with deltas
- do not use with Functional Range indexes
- do not use with querying/OQL
The following conditions result in significant fragmentation and would make use of off-heap memory questionable:
- value size is non-uniform
- value sizes vary dramatically
- usage pattern involves many updates instead of creates & destroys/clear
- usage pattern involves updates that result in significant changes to the value size
We do recommend using off-heap memory for all or any of the following:
- do use if value size is uniform
- do use if usage pattern involves cycles of many creates followed by clear (or many destroys)
- do use if customer needs to scale to very large memory
8.1 Sizing Recommendations
TODO: We need to determine the heap foot-print of off-heap values and then calculate some meaningful guidelines for sizing GemFire servers, a reasonable minimum overall off-heap-memory-size, minimum entry value size (for example: 3 byte value sizes would not be a good match for off-heap), etc.
8.1.1 Sizing Details
type | 32-bit JVM | 64-bit JVM (< 32 GB Java heap) | 64-bit JVM (> 32 GB Java heap) |
---|---|---|---|
ordinary object pointer | 4 bytes | 4 bytes (compressed OOPS) | 8 bytes |
object header | 8 bytes | 12 bytes | 16 bytes |
primitive long | 8 bytes | 8 bytes | 8 bytes |
object Long | 16 bytes | 24 bytes | 24 bytes |
The 64-bit JVM with compressed OOPS (enabled if Java heap is < 32 GB) performs better than 64-bit JVM > 32 GB Java heap. The 32-bit JVM is limited to 2-4 GB Java heap, so the sweet spot for a customer is to use the 64-bit JVM with < 32 GB Java heap and utilize our MemScale off-heap memory to scale as large as they need.