...
(4) RLCompressionExpScripts.sh:在不同的压缩方式参数下(UNCOMPRESSED, SNAPPY, GZIP, LZ4),写TsFile,清空系统缓存,然后进行读TsFile实验。LZ4),写TsFile,清空系统缓存,然后进行若干次重复读TsFile实验,把读实验结果进行汇总,把写文件的空间结果和读文件的耗时结果汇总到一起,最后对读实验结果进行平均值和百分比统计计算。
输入:
输出:不同压缩方式下的一个TsFile文件、一个TsFile空间统计结果文件( *writeResult.csv
)、REPEAT个读TsFile耗时结果csv文件( *readResult-T*csv
)、一个把重复读实验结果横向拼接起来的csv文件(*readResult-combined.csv
)、一个把写结果和读结果拼接起来的csv文件(*allResult-combined.csv
)、一个把读结果取平均值并且按照不同粒度统计百分比的csv文件( *allResult-combined-processed.csv
)
...
ZT11529传感器数据如下图所示,共12,780,287个点。
![](/confluence/download/attachments/228494026/image2022-9-14_23-20-35.png?version=1&modificationDate=1663168840000&api=v2)
压缩方式 | GZIP | LZ4 | SNAPPY | UNCOMPRESSED |
dataset | /disk/rl/zc_data/ZT11529.csv |
synthetic | synthetic | /disk/rl/zc_data/ZT11529.csv | /disk/rl/zc_data/ZT11529.csv | /disk/rl/zc_data/ZT11529.csv |
synthetic |
pagePointNum(ppn) | 10000 | 10000 | 10000 | 10000 |
numOfPagesInChunk(pic) | 100 |
10001000100010101013 |
timeEncoding(te) | TS_2DIFF | TS_2DIFF | TS_2DIFF | TS_2DIFF |
valueDataType(vt) | DOUBLE |
INT64INT64INT64DOUBLE |
valueEncoding(ve) | GORILLA |
PLAINPLAINPLAINGORILLA |
compression(co) | GZIP | LZ4 | SNAPPY | UNCOMPRESSED |
totalPointNum | 12780287 |
10000000010000000010000000012780287 |
tsfileSize(MB) | 19.34862614 |
77084443197679423904781422615130773735 |
chunkDataSize_stats_mean(MB) | 1.515139659 |
77084364367679412647814216614837062438 |
compressedPageSize_stats_mean(B) | 15824.16667 |
807648144480460477898187429684.7575 |
uncompressedPageSize_stats_mean(B) | 29684.7575 |
81874 | 81874 | 29684.7575 | 29684.7575 | 29684.7575 |
81874 |
timeBufferSize_stats_mean(B) | 11461.4625 |
1872 | 1872 | 11461.4625 | 11461.4625 | 11461.4625 |
1872 |
valueBufferSize_stats_mean(B) | 18221.26833 |
80000 | 80000 | 18221.26833 | 18221.26833 | 18221.26833 |
80000 |
[1] each step |
[Avg&Per] (A)1_index_read_deserialize_MagicString_FileMetadataSize(us) |
266428733 24422388846191903%119186528 016062988779113208%101882737 01309325873262339%109538906 014619657707769018%4111012934589997% |
[Avg&Per] (A)2_index_read_deserialize_IndexRootNode_MetaOffset_BloomFilter(us) |
57777715 05296237408352104%54849663 007392190510886776%51409507 0660679126109081%62195857 08300997092132265%6733833355290506% |
[Avg&Per] (A)3_2_index_read_deserialize_IndexRootNode_exclude_to_TimeseriesMetadata_forExactGet(us) |
692341118 06346396579532295%693314945 09343933722044904%675892748 08686102165735712%767226646 10239823783523703%251214480330036% |
[Avg&Per] (B)4_data_read_deserialize_ChunkHeader(us) |
86847625 07960952425196037%10008712599999999 13488927052826724%44879059 05767543633654741%70690819 09434780888370882%25653348589027036% |
[Avg&Per] (B)5_data_read_ChunkData(us) |
59409091292 54457787348789346%48396214844 6522447369141621%50821302731 6531199351132103%48443174031 6465489281142766%04477237472181% |
[Avg&Per] (C)6_data_deserialize_PageHeader(us) |
6613381399999991 060622054656159316%7120158900000014 09595969816001626%7692346500000014 09885667184764595%7605696300000007 10150975630087579%3649635223062446% |
[Avg&Per] (D-1)7_data_decompress_PageData(us) |
28594280031000106 26211160404158996%5218042723000004 7032452670194604%6052028143000009 7777644443672276%49817042259999976 6648853201570805%4515277590088287% |
[Avg&Per] (D-2)8_data_decode_PageData(us) |
1991910319299996 1825899474764487%19546574198999994 26343279504596413%19988805966999987 2568821922031179%20415179070999995 27247207495465567%54650374875476% |
[2] category: (A)get ChunkStatistic->(B)load on-disk Chunk->(C)get PageStatistics->(D)load in-memory PageData |
[Avg&Per] (A)get_chunkMetadatas |
1016547566 09318259204986696%867351136 11689451651044902%8291849919999999 10656107165107132%938961409 12531889263513831%335699109318087% |
[Avg&Per] (B)load_on_disk_chunk |
59495938917000005 5453739687304131%4849630197000001 6535936296194448%5086618179 6536966894765757%4851386484999999 6474924062031137%301305860612082% |
[Avg&Per] (C)get_pageHeader |
6613381399999991 060622054656159316%7120158900000014 09595969816001626%7692346500000014 09885667184764595%7605696300000007 10150975630087579%3649635223062446% |
[Avg&Per] (D_1)decompress_pageData |
28594280031000106 26211160404158996%5218042723000004 7032452670194604%6052028143000009 7777644443672276%49817042259999976 6648853201570805%4515277590088287% |
[Avg&Per] (D_2)decode_pageData |
1991910319299996 1825899474764487%19546574198999994 26343279504596413%19988805966999987 2568821922031179%20415179070999995 27247207495465567%54650374875476% |
[3] D_1 compare each step inside |
[Avg&Per] (D-1)7_1_data_ByteBuffer_to_ByteArray(us) |
65952.37819999998 us - 2.5136549346918415% | 108809.34350000018 us - 59.6896105506002% | 108132.35939999981 us - 43.622622294156905% | 110765.11740000003 us - 63.813731447511664% |
|
|
|
|
[Avg&Per] (D-1)7_2_data_decompress_PageDataByteArray(us) |
|
|
|
|
[Avg&Per] (D-1)7_ |
2decompressPageDataByteArray2554687.926599999 us - 97.36728361519128% | 68904.38600000006 us - 37.79892271446546% | 135345.91170000008 us - 54.601079805416944% | 57547.215800000035 us - 33.15396273496119% |
|
|
|
|
[Avg&Per] (D-1)7_4_data_split_time_value_Buffer(us) |
|
|
|
|
[3] D_2 compare each step inside |
[Avg&Per] (D-1)7_3_data_ByteArray_to_ByteBuffer(us) | 811.8335000000624 us - 0.03094155721322126% | 1239.949800000022 us - 0.680200047933345% | 1184.7460000000272 us - 0.47794876167766753% | 1229.1439000000355 us - 0.7081314098345313% |
[Avg&Per] (D-1)7_4_data_split_time_value_Buffer(us) | 2312.0582000000422 us - 0.08811989290364733% | 3338.2513999999974 us - 1.8312666870009688% | 3218.3657999999955 us - 1.29834913874849% | 4034.201500000018 us - 2.3241744076926314% |
[3] D_2 compare each step inside |
[Avg&Per] (D-2)8_1_createBatchData(us) | 5384.7852 us - 0.053292019060348375% | 5848.7599 us - 0.05759123169122766% | 5913.4963 us - 0.058362326692940975% | 6019.3023 us - 0.05943520403215091% |
[Avg&Per] (D-2)8_2_timeDecoder_hasNext(us) | 1859842.2956 us - 18.406444711361424% | 1862234.7849 us - 18.336946086748988% | 1864092.3926 us - 18.397368271414525% | 1857778.6739 us - 18.343895858133802% |
[Avg&Per] (D-2)8_3_timeDecoder_readLong(us) | 2074757.7936 us - 20.533415498567617% | 2084700.4377 us - 20.527508047369906% | 2063043.8916 us - 20.360888969091857% | 2069930.4964 us - 20.43870456313607%4_valueDecoder_read(us)1876012.952 us - 18.56648209392724% | 1881471.5433999998 us - 18.526365490982297% | 1877809.2412 us - 18.532744562964893% | 1876843.1276 us - 18.53214021585961%1_createBatchData(us) |
|
|
|
|
[Avg&Per] (D-2)8_ |
5checkValueSatisfyOrNot1780379.6374 us - 17.620020492363725% | 1780782.3133 us - 17.534904586680103% | 1781949.2049 us - 17.586668929952697% |
|
|
|
|
[Avg&Per] (D-2)8_3_timeDecoder_readLong(us) |
|
|
|
|
[Avg&Per] (D-2)8_4_valueDecoder_read(us) |
|
|
|
|
[Avg&Per] (D-2)8_5_checkValueSatisfyOrNot(us) |
1780599.5789 us - 17.5818216126948% |
|
|
|
|
[Avg&Per] (D-2)8_6_putIntoBatchData(us) |
2507922.0072 us - 24.82034518471963% | 2540605.1784 us - 25.016684556527476% | 2539577.912 us - 25.063966939883077% | 2536332.2055 us - 25.044002546143567%
改变编码方式
人工数据实验结果
中车数据实验结果