Ignite WAL reader

This article describes only MVP/prototype of WAL reader, usage details will be changed

General description

Ignite WAL reader is console utility prototype for Ignite developers. This utility will allow developers to check WAL content and perform WAL analysis.

Utility converts WAL binary files into human readable records and dumps it to StdOut.

The IDEA of this utility is very simple - get the possibility to look into your WAL.

But it also requires understanding what's going on in WAL mechanism (core WAL record description can be found in Ignite Persistent Store - under the hood in section WAL structure )

Usage

Some tips how to use converter:
Example of command line:

java -cp "lib/ignite-dev-utils-2.4.0-SNAPSHOT-jar-with-dependencies.jar" -DIGNITE_PDS_SKIP_CRC=true -DPRINT_STAT=true -DPRINT_RECORDS=false org.apache.ignite.development.utils.IgniteWalConverter %*

Alternatively if separate JAR files are used following

java -cp "lib/ignite-dev-utils-2.3.0-SNAPSHOT.jar:lib/ignite-indexing-2.3.0-SNAPSHOT.jar:lib/ignite-core-2.3.0-SNAPSHOT.jar:~/.m2/repository/com/h2database/h2/1.4.195/h2-1.4.195.jar" org.apache.ignite.development.utils.IgniteWalConverter 4096 ~/IdeaProjects/incubator-ignite/work/db/wal/127_0_0_1_47503 ~/IdeaProjects/incubator-ignite/work/db/wal/archive/127_0_0_1_47503

You need to pass as argument:
1. Page size.
2. Path to WAL dir.
3. (Optional) Path to Archive WAL dir.

Record output

Format of record output

To enable record output please set java system property: PRINT_RECORDS=true

Output consist from lines, every new record discovered causes new line

(Record source) (record string representation)

Record source - W - Work directory, A - Archive directory

Record string representation starts from record type name. Then record parameters are dumped. Record dump is determined by record class itself.

Note record may provide not all, but only printable human readable details. Record may hide long binary objects which non appropriate for human readable form.

Example of records output

To view example of output click below.

Click here to expand...

[W] InitNewPageRecord [newPageId=0001001c00000067, super=PageDeltaRecord
[grpId=2141373875, pageId=0001001c00000067, super=WALRecord [size=41,
chainSize=0, pos=FileWALPointer [idx=1, fileOffset=25406, len=41,
forceFlush=false], type=INIT_NEW_PAGE_RECORD]]]
[W] InitNewPageRecord [newPageId=0001001c00000068, super=PageDeltaRecord
[grpId=2141373875, pageId=0001001c00000068, super=WALRecord [size=41,
chainSize=0, pos=FileWALPointer [idx=1, fileOffset=25447, len=41,
forceFlush=false], type=INIT_NEW_PAGE_RECORD]]]
[W] PagesListAddPageRecord [dataPageId=0001001c0000002a,
super=PageDeltaRecord [grpId=2141373875, pageId=0001001c00000068,
super=WALRecord [size=37, chainSize=0, pos=FileWALPointer [idx=1,
fileOffset=25488, len=37, forceFlush=false], type=PAGES_LIST_ADD_PAGE]]]
[W] PageSnapshot [fullPageId = FullPageId [pageId=0001001c0000002a,
effectivePageId=0000001c0000002a, grpId=2141373875], page = [
Header [
type=1 (DataPageIO),
ver=1,
crc=0,
pageId=281595235794986(offset=0, flags=1, partId=28, index=42)
],
DataPageIO [
0001001c0000002a [4049, 4002, 3955, 3908, 3861, 3814, 3767, 3720, 3673,
3626][][free=3552, actualFree=-544
]],
super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
fileOffset=25525, len=4125, forceFlush=false], type=PAGE_RECORD]]]
[W] InsertRecord [idx=12, io=CacheIdAwareDataLeafIO[ver=1],
rightId=0000000000000000, super=PageDeltaRecord [grpId=2141373875,
pageId=0001001c00000004, super=WALRecord [size=59, chainSize=0,
pos=FileWALPointer [idx=1, fileOffset=29650, len=59, forceFlush=false],
type=BTREE_PAGE_INSERT]]]
[W] DataRecord [writeEntries=[DataEntry [cacheId=-1368047377, op=CREATE,
writeVer=GridCacheVersion [topVer=116892426, order=1505412445450,
nodeOrder=2], partId=28, partCnt=69]], super=WALRecord [size=100,
chainSize=0, pos=FileWALPointer [idx=1, fileOffset=29709, len=100,
forceFlush=false], type=DATA_RECORD]]
[W] PageSnapshot [fullPageId = FullPageId [pageId=000100000000000a,
effectivePageId=000000000000000a, grpId=2141373875], page = [
Header [
type=20 (PagePartitionCountersIO),
ver=1,
crc=0,
pageId=281474976710666(offset=0, flags=1, partId=0, index=10)
],
PagePartitionCounters [
count=1,
lastFlag=true,
nextCountersPageId=0000000000000000,
size={
-1368047377=313
}
]],
super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
fileOffset=29809, len=4125, forceFlush=false], type=PAGE_RECORD]]]
[W] PageSnapshot [fullPageId = FullPageId [pageId=0001000200000010,
effectivePageId=0000000200000010, grpId=2141373875], page = [
Header [
type=20 (PagePartitionCountersIO),
ver=1,
crc=0,
pageId=281483566645264(offset=0, flags=1, partId=2, index=16)
],
PagePartitionCounters [
count=1,
lastFlag=true,
nextCountersPageId=0000000000000000,
size={
-1368047377=313
}
]],
super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
fileOffset=33934, len=4125, forceFlush=false], type=PAGE_RECORD]]]
[W] PageSnapshot [fullPageId = FullPageId [pageId=000100050000000d,
effectivePageId=000000050000000d, grpId=2141373875], page = [
Header [
type=20 (PagePartitionCountersIO),
ver=1,
crc=0,
pageId=281496451547149(offset=0, flags=1, partId=5, index=13)
],
PagePartitionCounters [
count=1,
lastFlag=true,
nextCountersPageId=0000000000000000,
size={
-1368047377=313
}
]],
super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
fileOffset=38059, len=4125, forceFlush=false], type=PAGE_RECORD]]]
[W] PageSnapshot [fullPageId = FullPageId [pageId=0001000900000003,
effectivePageId=0000000900000003, grpId=2141373875], page = [
Header [
type=12 (PagesListMetaIO),
ver=1,
crc=0,
pageId=281513631416323(offset=0, flags=1, partId=9, index=3)
],
PagesListMeta [
nextMetaPageId=0000000000000000,
count=230,
bucketData={
bucket=163, list=GridLongList [idx=6,
arr=[281513631416558,281513631416559,281513631416560,281513631416561,281513631416562,281513631416563]]
bucket=166, list=GridLongList [idx=8,
arr=[281513631416550,281513631416551,281513631416552,281513631416553,281513631416554,281513631416555,281513631416556,281513631416557]]
bucket=169, list=GridLongList [idx=8,
arr=[281513631416542,281513631416543,281513631416544,281513631416545,281513631416546,281513631416547,281513631416548,281513631416549]]
bucket=172, list=GridLongList [idx=8,
arr=[281513631416534,281513631416535,281513631416536,281513631416537,281513631416538,281513631416539,281513631416540,281513631416541]]
bucket=175, list=GridLongList [idx=8,
arr=[281513631416526,281513631416527,281513631416528,281513631416529,281513631416530,281513631416531,281513631416532,281513631416533]]
bucket=178, list=GridLongList [idx=8,
arr=[281513631416512,281513631416513,281513631416520,281513631416521,281513631416522,281513631416523,281513631416524,281513631416525]]
bucket=181, list=GridLongList [idx=8,
arr=[281513631416510,281513631416511,281513631416514,281513631416515,281513631416516,281513631416517,281513631416518,281513631416519]]
bucket=184, list=GridLongList [idx=8,
arr=[281513631416500,281513631416503,281513631416504,281513631416505,281513631416506,281513631416507,281513631416508,281513631416509]]
bucket=187, list=GridLongList [idx=8,
arr=[281513631416494,281513631416495,281513631416496,281513631416497,281513631416498,281513631416499,281513631416501,281513631416502]]
bucket=190, list=GridLongList [idx=8,
arr=[281513631416484,281513631416485,281513631416486,281513631416487,281513631416488,281513631416489,281513631416492,281513631416493]]
bucket=193, list=GridLongList [idx=8,
arr=[281513631416448,281513631416449,281513631416480,281513631416481,281513631416482,281513631416483,281513631416490,281513631416491]]
bucket=196, list=GridLongList [idx=8,
arr=[281513631416446,281513631416447,281513631416474,281513631416475,281513631416476,281513631416477,281513631416478,281513631416479]]
bucket=199, list=GridLongList [idx=8,
arr=[281513631416444,281513631416445,281513631416468,281513631416469,281513631416470,281513631416471,281513631416472,281513631416473]]
bucket=202, list=GridLongList [idx=8,
arr=[281513631416436,281513631416437,281513631416462,281513631416463,281513631416464,281513631416465,281513631416466,281513631416467]]
bucket=205, list=GridLongList [idx=8,
arr=[281513631416434,281513631416435,281513631416456,281513631416457,281513631416458,281513631416459,281513631416460,281513631416461]]
bucket=209, list=GridLongList [idx=8,
arr=[281513631416426,281513631416427,281513631416450,281513631416451,281513631416452,281513631416453,281513631416454,281513631416455]]
bucket=212, list=GridLongList [idx=8,
arr=[281513631416418,281513631416419,281513631416438,281513631416439,281513631416440,281513631416441,281513631416442,281513631416443]]
bucket=215, list=GridLongList [idx=8,
arr=[281513631416416,281513631416417,281513631416428,281513631416429,281513631416430,281513631416431,281513631416432,281513631416433]]
bucket=218, list=GridLongList [idx=8,
arr=[281513631416414,281513631416415,281513631416420,281513631416421,281513631416422,281513631416423,281513631416424,281513631416425]]
bucket=221, list=GridLongList [idx=8,
arr=[281513631416406,281513631416407,281513631416408,281513631416409,281513631416410,281513631416411,281513631416412,281513631416413]]
bucket=224, list=GridLongList [idx=8,
arr=[281513631416398,281513631416399,281513631416400,281513631416401,281513631416402,281513631416403,281513631416404,281513631416405]]
bucket=227, list=GridLongList [idx=8,
arr=[281513631416390,281513631416391,281513631416392,281513631416393,281513631416394,281513631416395,281513631416396,281513631416397]]
bucket=230, list=GridLongList [idx=8,
arr=[281513631416381,281513631416382,281513631416383,281513631416384,281513631416385,281513631416386,281513631416387,281513631416388]]
bucket=233, list=GridLongList [idx=8,
arr=[281513631416373,281513631416374,281513631416375,281513631416376,281513631416377,281513631416378,281513631416379,281513631416380]]
bucket=236, list=GridLongList [idx=8,
arr=[281513631416365,281513631416366,281513631416367,281513631416368,281513631416369,281513631416370,281513631416371,281513631416372]]
bucket=239, list=GridLongList [idx=8,
arr=[281513631416351,281513631416352,281513631416359,281513631416360,281513631416361,281513631416362,281513631416363,281513631416364]]
bucket=242, list=GridLongList [idx=8,
arr=[281513631416348,281513631416349,281513631416353,281513631416354,281513631416355,281513631416356,281513631416357,281513631416358]]
bucket=245, list=GridLongList [idx=8,
arr=[281513631416335,281513631416336,281513631416337,281513631416338,281513631416339,281513631416340,281513631416346,281513631416347]]
bucket=248, list=GridLongList [idx=8,
arr=[281513631416326,281513631416327,281513631416328,281513631416329,281513631416330,281513631416331,281513631416343,281513631416344]]
}
]],
super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
fileOffset=42184, len=4125, forceFlush=false], type=PAGE_RECORD]]]
[W] PageSnapshot [fullPageId = FullPageId [pageId=000100090000000d,
effectivePageId=000000090000000d, grpId=2141373875], page = [
Header [
type=20 (PagePartitionCountersIO),
ver=1,
crc=0,
pageId=281513631416333(offset=0, flags=1, partId=9, index=13)
],
PagePartitionCounters [
count=1,
lastFlag=true,
nextCountersPageId=0000000000000000,
size={
-1368047377=198
}
]],
super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
fileOffset=46309, len=4125, forceFlush=false], type=PAGE_RECORD]]]
[W] PageSnapshot [fullPageId = FullPageId [pageId=0001000900000000,
effectivePageId=0000000900000000, grpId=2141373875], page = [
Header [
type=14 (PagePartitionMetaIO),
ver=1,
crc=0,
pageId=281513631416320(offset=0, flags=1, partId=9, index=0)
],
PageMeta[
treeRoot=281513631416323,
lastSuccessfulFullSnapshotId=0,
lastSuccessfulSnapshotId=0,
nextSnapshotTag=1,
lastSuccessfulSnapshotTag=0,
lastAllocatedPageCount=0,
candidatePageCount=0
],
PagePartitionMeta[
size=198,
updateCounter=198,
globalRemoveId=1505412422861000000,
partitionState=1(OWNING),
countersPageId=281513631416333
]],
super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
fileOffset=50434, len=4125, forceFlush=false], type=PAGE_RECORD]]]
[W] PageSnapshot [fullPageId = FullPageId [pageId=0001000b0000000c,
effectivePageId=0000000b0000000c, grpId=2141373875], page = [
Header [
type=20 (PagePartitionCountersIO),
ver=1,
crc=0,
pageId=281522221350924(offset=0, flags=1, partId=11, index=12)
],
PagePartitionCounters [
count=1,
lastFlag=true,
nextCountersPageId=0000000000000000,
size={
-1368047377=313
}
]],
super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
fileOffset=54559, len=4125, forceFlush=false], type=PAGE_RECORD]]]
[W] PageSnapshot [fullPageId = FullPageId [pageId=0001000c0000000f,
effectivePageId=0000000c0000000f, grpId=2141373875], page = [
Header [
type=20 (PagePartitionCountersIO),
ver=1,
crc=0,
pageId=281526516318223(offset=0, flags=1, partId=12, index=15)
],
PagePartitionCounters [
count=1,
lastFlag=true,
nextCountersPageId=0000000000000000,
size={
-1368047377=313
}
]],
super = [WALRecord [size=4125, chainSize=0, pos=FileWALPointer [idx=1,
fileOffset=58684, len=4125, forceFlush=false], type=PAGE_RECORD]]]

References

http://apache-ignite-developers.2346864.n4.nabble.com/Ignite-PDS-WAL-analysis-with-human-readable-records-td22277.html

Unable to render Jira issues macro, execution error.

Page tree