...
Table of Contents:
Table of Contents |
---|
There are following file types used for persisting data: Cache pages or page store, Checkpoint markers, and WAL segments
Ignite with enabled persistence uses following folder stucture
Consistent ID may be configured using IgniteConfiguration or generated from local IPs set by default.
Partitions of each cache have corresponding file in page store directory (particular node may own not all partitions).
Special partition 65535 is used for SQL indexes and saved to index.bin
...
Practically we can’t replay WAL from the beginning of times, Volume(HDD)<Volume(full WAL), and we need procedure to throw out oldest part of changes in WAL.
Consistent state comes only from pair of WAL and page store.
This procedure is named checkpointing
...
Implemented - Sharp Checkpoint; F.C. - todoto be done in future releases.
To achieve consistency Checkpoint read-write lock is used (see GridCacheDatabaseSharedManager#checkpointLock)
...
Dirty pages is set, when page from non-dirty becomes dirty, it is added to this set.
Collection of pages (GridCacheDatabaseSharedManager.Checkpoint#cpPages) allows us to collect and then write pages which were changed since last checkpoint.
...
...
...
...
Crash recovery involves following records writtent in WAL, it may be of 2 main types
For particular cache entry update we log records in follwowing order:
...
Possible future optimisation - refer data modified from PageDeltaRecord to logical record. Will allow to not store byte updates twice. There is file WAL pointer, pointer to record from the beginning of time. This refreence may be used.
WAL file segments and rotation structure
See also WAL history size section below
Let’s assume node start process is running with existent files.
...
If transaction begin record has no corresponding end, tx change is not applied.
Ignite with enabled persistence uses following folder stucture
Consistent ID may be configured using IgniteConfiguration or generated from local IPs set
There are following file types used for persisting data: Cache pages or page store, Checkpoint markers, and WAL segments
Consistent state comes only from pair of WAL and page store.
Because CP are consistent we can’t start next CP until previous is not completed.
...
There several levels of guarantees (WALMode)
Implementation | WarannyWarranties | |
---|---|---|
DEFAULT | fsync() on each commit | Any crashes (OS and process crash) |
LOG_ONY | write() on commit Synchronisation is responsibility of OS | Kill process, but no OS fail |
BACKGROUND | do nothing on commit (records are accumulated in memory) write() on timeout | kill -9 may cause loss of several latest updates |
...