THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
压缩方式 | GZIP | LZ4 | SNAPPY | UNCOMPRESSED |
dataset | /disk/rl/zc_data/ZT11529.csv | /disk/rl/zc_data/ZT11529.csv | /disk/rl/zc_data/ZT11529.csv | /disk/rl/zc_data/ZT11529.csv |
pagePointNum(ppn) | 10000 | 10000 | 10000 | 10000 |
numOfPagesInChunk(pic) | 100 | 100 | 100 | 100 |
chunksWritten(cw) | 13 | 13 | 13 | 13 |
timeEncoding(te) | TS_2DIFF | TS_2DIFF | TS_2DIFF | TS_2DIFF |
valueDataType(vt) | DOUBLE | DOUBLE | DOUBLE | DOUBLE |
valueEncoding(ve) | GORILLA | GORILLA | GORILLA | GORILLA |
compression(co) | GZIP | LZ4 | SNAPPY | UNCOMPRESSED |
totalPointNum | 12780287 | 12780287 | 12780287 | 12780287 |
tsfileSize(MB) | 19.34862614 | 23.77741051 | 23.15641212 | 36.30773735 |
chunkDataSize_stats_mean(MB) | 1.515139659 | 1.86007007 | 1.813184341 | 2.837062438 |
compressedPageSize_stats_mean(B) | 15824.16667 | 19440.275 | 18948.69 | 29684.7575 |
uncompressedPageSize_stats_mean(B) | 29684.7575 | 29684.7575 | 29684.7575 | 29684.7575 |
timeBufferSize_stats_mean(B) | 11461.4625 | 11461.4625 | 11461.4625 | 11461.4625 |
valueBufferSize_stats_mean(B) | 18221.26833 | 18221.26833 | 18221.26833 | 18221.26833 |
[2] category: (A)get ChunkStatistic->(B)load on-disk Chunk->(C)get PageStatistics->(D)load in-memory PageData | ||||
[Avg&Per] (A)get_chunkMetadatas | 87316.85010000001 us - 8.19883087518919% | 100289.6416 us - 12.59103802007131% | 89466.6045 us - 11.199530364576052% | 88760.777 us - 10.335699109318087% |
[Avg&Per] (B)load_on_disk_chunk | 160699.6025 us - 15.089285299443361% | 176784.88239999997 us - 22.19476647997349% | 105802.9384 us - 13.244531050378352% | 191518.85280000002 us - 22.301305860612082% |
[Avg&Per] (C)get_pageHeader | 2436.5129999999995 us - 0.22878239411203669% | 2198.3668000000007 us - 0.27599779517870476% | 2319.9517000000005 us - 0.29041416798711567% | 3134.228800000001 us - 0.3649635223062446% |
[Avg&Per] (D_1)decompress_pageData | 356587.5454999999 us - 33.482666568996265% | 31158.160799999983 us - 3.911805656191469% | 115629.7179 us - 14.474658381255693% | 29640.983400000016 us - 3.4515277590088287% |
[Avg&Per] (D_2)decode_pageData | 457950.96670000016 us - 43.000434862259155% | 486085.0215000002 us - 61.02639204858503% | 485623.25309999986 us - 60.790866035802786% | 545723.8052999998 us - 63.54650374875476% |
SUM | 1064991.4778 us | 796516.0731000002 us | 798842.4655999999 us | 858778.6472999998 us |
[3] D_1 compare each step inside | ||||
[Avg&Per] (D-1)7_1_data_ByteBuffer_to_ByteArray(us) | 1269.7615999999996 us - 0.36377892330196115% | 1808.5776999999991 us - 7.207505165727991% | 1687.9916000000003 us - 1.412377327892232% | 3197.6313 us - 41.86667970732365% |
[Avg&Per] (D-1)7_2_data_decompress_PageDataByteArray(us) | 345856.23130000004 us - 99.08569249502276% | 21247.39440000001 us - 84.67466169480036% | 116100.69619999993 us - 97.14384305311928% | 2432.4552000000012 us - 31.84820675254647% |
[Avg&Per] (D-1)7_3_data_ByteArray_to_ByteBuffer(us) | 374.18470000000025 us - 0.10720162531460037% | 442.10720000000003 us - 1.761876157051776% | 424.26030000000026 us - 0.35498732863644405% | 421.1930000000002 us - 5.51469221169019% |
[Avg&Per] (D-1)7_4_data_split_time_value_Buffer(us) | 1547.4220999999993 us - 0.443326956360674% | 1594.8989000000004 us - 6.355956982419887% | 1301.2614999999998 us - 1.0887922903520593% | 1586.372500000001 us - 20.770421328439692% |
[3] D_2 compare each step inside | ||||
[Avg&Per] (D-2)8_1_createBatchData(us) | 3430.8672 us - 0.22855030377902266% | 4174.8958 us - 0.275155541260775% | 3438.1465 us - 0.22874736939899784% | 3730.6205 us - 0.24721094086707432% |
[Avg&Per] (D-2)8_2_timeDecoder_hasNext(us) | 234016.77980000002 us - 15.589238228946503% | 236135.9462 us - 15.56305048087338% | 235278.4135 us - 15.65358490817499% | 234469.6457 us - 15.537217392727715% |
[Avg&Per] (D-2)8_3_timeDecoder_readLong(us) | 357893.9434 us - 23.841426880277478% | 360253.1425 us - 23.743262865502565% | 358341.1524 us - 23.84121675993312% | 363063.0738 us - 24.058508247673558% |
[Avg&Per] (D-2)8_4_valueDecoder_read(us) | 353821.6809 us - 23.570149451806063% | 359477.8899 us - 23.692168165422427% | 356440.4841 us - 23.714761161335133% | 355773.1939 us - 23.5754416723178% |
[Avg&Per] (D-2)8_5_checkValueSatisfyOrNot(us) | 223758.3096 us - 14.905861011513531% | 224938.2721 us - 14.825043539994219% | 224282.1638 us - 14.921980483485841% | 226053.6587 us - 14.979528915812129% |
[Avg&Per] (D-2)8_6_putIntoBatchData(us) | 328221.5562 us - 21.864774123677403% | 332305.597 us - 21.90131940694663% | 325251.7878 us - 21.639709317671908% | 325993.7043 us - 21.602092830601723% |
...
编码方式 | GORILLA | PLAIN | RLE | TS_2DIFF |
dataset | /disk/rl/zc_data/ZT11529.csv | /disk/rl/zc_data/ZT11529.csv | /disk/rl/zc_data/ZT11529.csv | /disk/rl/zc_data/ZT11529.csv |
pagePointNum(ppn) | 10000 | 10000 | 10000 | 10000 |
numOfPagesInChunk(pic) | 100 | 100 | 100 | 100 |
chunksWritten(cw) | 13 | 13 | 13 | 13 |
timeEncoding(te) | TS_2DIFF | TS_2DIFF | TS_2DIFF | TS_2DIFF |
valueDataType(vt) | DOUBLE | DOUBLE | DOUBLE | DOUBLE |
valueEncoding(ve) | GORILLA | PLAIN | RLE | TS_2DIFF |
compression(co) | SNAPPY | SNAPPY | SNAPPY | SNAPPY |
totalPointNum | 12780287 | 12780287 | 12780287 | 12780287 |
tsfileSize(MB) | 23.15641212 | 26.59960175 | 22.10147858 | 20.98550797 |
chunkDataSize_stats_mean(MB) | 1.813184341 | 2.081166744 | 1.740072568 | 1.642661015 |
compressedPageSize_stats_mean(B) | 18948.69 | 21758.61667 | 18182.29917 | 17160.8925 |
uncompressedPageSize_stats_mean(B) | 29684.7575 | 91463.48917 | 25245.93417 | 22210.36417 |
timeBufferSize_stats_mean(B) | 11461.4625 | 11461.4625 | 11461.4625 | 11461.4625 |
valueBufferSize_stats_mean(B) | 18221.26833 | 80000 | 13782.445 | 10746.875 |
[2] category: (A)get ChunkStatistic->(B)load on-disk Chunk->(C)get PageStatistics->(D)load in-memory PageData | ||||
[Avg&Per] (A)get_chunkMetadatas | 88753.03020000001 us - 11.21274000219672% | 97256.88870000001 us - 12.181751942467722% | 84509.5108 us - 9.498496964807579% | 86370.7961 us - 11.6687579647861% |
[Avg&Per] (B)load_on_disk_chunk | 105215.9241 us - 13.29260306229143% | 138228.6108 us - 17.313597737139155% | 139816.5536 us - 15.714765088895252% | 96826.5282 us - 13.081335047881257% |
[Avg&Per] (C)get_pageHeader | 2352.1583 us - 0.2971632563133493% | 2625.5312999999983 us - 0.328856613051254% | 3388.6254999999987 us - 0.38086658793698225% | 3159.255499999999 us - 0.4268177375109232% |
[Avg&Per] (D_1)decompress_pageData | 114457.26139999992 us - 14.460120522641777% | 182862.35910000015 us - 22.904124612107367% | 155021.76800000004 us - 17.42376424722015% | 120684.50350000002 us - 16.304565026949902% |
[Avg&Per] (D_2)decode_pageData | 480759.0216999998 us - 60.73737315655672% | 377408.39589999994 us - 47.27166909523449% | 506978.10549999995 us - 56.982107111140046% | 433147.3443000001 us - 58.51852422287182% |
SUM | 791537.3956999998 us | 798381.7858000002 us | 889714.5634 us | 740188.4276000002 |
[3] D_1 compare each step inside | ||||
[Avg&Per] (D-1)7_1_data_ByteBuffer_to_ByteArray(us) | 1754.3673999999999 us - 1.6860113426598728% | 2320.3632000000002 us - 1.489795195953362% | 1825.1960999999988 us - 1.548613686329812% | 1607.3271999999997 us - 1.758246074195737% |
[Avg&Per] (D-1)7_2_data_decompress_PageDataByteArray(us) | 100527.56879999992 us - 96.61067644486589% | 151532.73299999998 us - 97.29198327791244% | 113686.73430000001 us - 96.4591326329927% | 87212.82 us - 95.4016073295714% |
[Avg&Per] (D-1)7_3_data_ByteArray_to_ByteBuffer(us) | 431.9958999999999 us - 0.41516388607230165% | 468.2119000000003 us - 0.30061666178303303% | 474.28859999999975 us - 0.4024169332984032% | 663.3730000000002 us - 0.7256599483773118% |
[Avg&Per] (D-1)7_4_data_split_time_value_Buffer(us) | 1340.373799999999 us - 1.288148326401935% | 1429.1740000000002 us - 0.9176048643511714% | 1873.7816000000007 us - 1.5898367473790769% | 1932.989900000001 us - 2.1144866478555286% |
[3] D_2 compare each step inside | ||||
[Avg&Per] (D-2)8_1_createBatchData(us) | 3348.9259 us - 0.2218271110091239% | 3442.3069 us - 0.24318767981262365% | 3466.6887 us - 0.22277034162258696% | 3338.6453 us - 0.22194982158403212% |
[Avg&Per] (D-2)8_2_timeDecoder_hasNext(us) | 235317.741 us - 15.587043790733997% | 244839.8323 us - 17.29713023052909% | 239308.8755 us - 15.37805224577913% | 240223.005 us - 15.969786637750962% |
[Avg&Per] (D-2)8_3_timeDecoder_readLong(us) | 361562.5392 us - 23.949282819264262% | 363364.4154 us - 25.67050285597614% | 362143.5815 us - 23.271443255687238% | 357833.1113 us - 23.788389623148674% |
[Avg&Per] (D-2)8_4_valueDecoder_read(us) | 356325.4946 us - 23.602389962111484% | 241625.5526 us - 17.07005192367858% | 396438.5566 us - 25.475247513902037% | 335421.7093 us - 22.298501890735768% |
[Avg&Per] (D-2)8_5_checkValueSatisfyOrNot(us) | 225554.2633 us - 14.940327764084076% | 232364.86419999998 us - 16.415814695306036% | 224510.63450000001 us - 14.4271133273284% | 230332.4743 us - 15.312273986066641% |
[Avg&Per] (D-2)8_6_putIntoBatchData(us) | 327591.9399 us - 21.699128552797042% | 329856.9807 us - 23.30331261469754% | 330303.223 us - 21.225373315680617% | 337085.3345 us - 22.409098040713936% |
补充实验
时间戳和值都使用PLAIN编码,然后改变压缩。
RLCompressionRealExpScripts.sh
压缩方式 | GZIP | LZ4 | SNAPPY | UNCOMPRESSED |
dataset | ||||
pagePointNum(ppn) | ||||
numOfPagesInChunk(pic) | ||||
chunksWritten(cw) | ||||
timeEncoding(te) | ||||
valueDataType(vt) | ||||
valueEncoding(ve) | ||||
compression(co) | ||||
totalPointNum | ||||
tsfileSize(MB) | ||||
chunkDataSize_stats_mean(MB) | ||||
compressedPageSize_stats_mean(B) | ||||
uncompressedPageSize_stats_mean(B) | ||||
timeBufferSize_stats_mean(B) | ||||
valueBufferSize_stats_mean(B) | ||||
[2] category: (A)get ChunkStatistic->(B)load on-disk Chunk->(C)get PageStatistics->(D)load in-memory PageData | ||||
[Avg&Per] (A)get_chunkMetadatas | ||||
[Avg&Per] (B)load_on_disk_chunk | ||||
[Avg&Per] (C)get_pageHeader | ||||
[Avg&Per] (D_1)decompress_pageData | ||||
[Avg&Per] (D_2)decode_pageData | ||||
SUM | ||||
[3] D_1 compare each step inside | ||||
[Avg&Per] (D-1)7_1_data_ByteBuffer_to_ByteArray(us) | ||||
[Avg&Per] (D-1)7_2_data_decompress_PageDataByteArray(us) | ||||
[Avg&Per] (D-1)7_3_data_ByteArray_to_ByteBuffer(us) | ||||
[Avg&Per] (D-1)7_4_data_split_time_value_Buffer(us) | ||||
[3] D_2 compare each step inside | ||||
[Avg&Per] (D-2)8_1_createBatchData(us) | ||||
[Avg&Per] (D-2)8_2_timeDecoder_hasNext(us) | ||||
[Avg&Per] (D-2)8_3_timeDecoder_readLong(us) | ||||
[Avg&Per] (D-2)8_4_valueDecoder_read(us) | ||||
[Avg&Per] (D-2)8_5_checkValueSatisfyOrNot(us) | ||||
[Avg&Per] (D-2)8_6_putIntoBatchData(us) |
人工数据实验结果
改变压缩方式
RLCompressionSynExpScripts.sh
...