Page History

...

In-memory caches store pages in configured DataRegion. Page for specific cache group allocated in some Segment of data region.

So, unlike persistent caches it more convinient and error-prone to create snapshot of the DataRegion with all caches in it.

During creation of snapshot node must track all page changes which can be implemented by the listener of write locks in PageMemoryNoStoreImplMoreover, pages of data region can store entries from different cache-groups.
So for in-memory caches snapshots it proposed to use per entry storage.
Every entry in cache group will be saved as is to corresponding file.

3. Persistent pages contains CRC to make sure data integrity while storing on disk:

...

must be properly prepared and saved during snapshot.

High-level snapshot creation algorithm steps:

1. Under PME locks setup page before writeLock listener.
   Listener must be invoked under writeLock of page but before code that trying to acquire lock.
2. Iterate each page of DataRegion and write page entries to corresponding cache-group snapshot files.
3. Inside listener:
   1. store pageId that was handled by listener
   2. handle page like on step2.

After algorithm ends each file will contain consistent copy of in-memory cache group.

Restore

Prerequisites:

• Restored data region is empty - there are no any caches stored in itcache group not exists on cluster.
• Count of nodes in cluster are the same as in time of creation (this restriction can be eliminated in Phase 2 of IEP implementation).
• All nodes in cluster has snapshot locally.

Steps:

High-level snapshot restore algorithm steps:

1. Create corresponding cachesBlock data region exclusively on each node - any attempt of usage (cache creation) must be blocked.
2. Restore all saved data into data regionmetadata.Restore all saved metadata
3. Iterate saved entries and put them as local cache entries.
4. Wait all nodes complete step 2 and 3.
5. Start caches that belongs to restored data region.

...

There are a couple of alternatives to implement backup copies of inmemory caches that was rejected during initial analyzes:

1. Store entries instead of full data region copy 
   The idea of this approach is to store entries in the file instead of pagesfull data region copy on the disk.
   
   1. Pros:
      1. cache group granularity like in persistent snapshots.
      2. smaller snapshot size in case of snapshotting specific cache group. Currently, cache group snapshot granularity not supported by persistent snapshots.
      3. backward compatibility of BinaryObject only required. PageIO strusture can be changed 
      4. ability to implement primary-only mode.
      5. Mostly sequential writes.
   2. Cons:
      1. restore must setup all Ignite internals strustured regarding restored offheap data.
         Such code can be difficult to implement and maintain
      Cons:
      1. restore require more time because per-entry local put operation must be invoked on each node.
2. On demand persistence
   The idea of this approach is to reuse PDS infrastucture and persistent snapshot code by introducing new cache mode "PDS_ON_DEMAND".
   This mode will use persitence code but with WAL and checkpoint disabled. So on creating snapshot regular checkpoint will be performed.
   After checkpoint PDS files are ready to be copied to snapshot folder.
   1. Pros:
      1. Code reuse.
   2. Cons:
      1. Additional configuration on user-side required (set new mode for DataRegion).
      2. All Ignite codebase needs to be aware of new mode - baseline, PDS, WAL, checkpoint, etc.
      3. PDS page stores additional data - storage overhead.
3. shmem usage
   The idea of this approach is to use shared memory feature of Linux.
   1. Pros:
      1. Easy to implement (questioneable).
      2. Same approach used by other vendors to implement in-memory cluster restarts.
   2. Cons:
      1. OS specific.
      2. Only for certain scenarios. Doesn't cover all use-cases.

...