一、实验设置
1 实验目的
- 读一个TsFile的磁盘IO代价和CPU代价的大小比较
- 了解CPU代价里有没有突出的耗时瓶颈操作
2 实验设计
...
分析TsFile文件结构,然后把读取TsFile的耗时步骤进行分解和分类,最后进行实验测量统计。
实验环境
2.1TsFile结构
Image Added
2.2 TsFile读耗时步骤分解与分类
Image Added
耗时分类 | 耗时分解 |
(A)get_chunkMetadatas | (A)1_index_read_deserialize_MagicString_FileMetadataSize |
(A)2_index_read_deserialize_IndexRootNode_MetaOffset_BloomFilter |
(A)3_2_index_read_deserialize_IndexRootNode_exclude_to_TimeseriesMetadata_forExactGet |
(B)load_on_disk_chunk | (B)4_data_read_deserialize_ChunkHeader |
(B)5_data_read_ChunkData |
(C)get_pageHeader | (C)6_data_deserialize_PageHeader |
(D-1)decompress_pageData | (D-1)7_1_data_ByteBuffer_to_ByteArray |
(D-1)7_2_data_decompress_PageDataByteArray |
(D-1)7_3_data_ByteArray_to_ByteBuffer |
(D-1)7_4_data_split_time_value_Buffer |
(D-2)decode_pageData | (D-2)8_1_createBatchData |
(D-2)8_2_timeDecoder_hasNext |
(D-2)8_3_timeDecoder_readLong |
(D-2)8_4_valueDecoder_read |
(D-2)8_5_checkValueSatisfyOrNot |
(D-2)8_6_putIntoBatchData |
回答实验目的1:分析A、B、C、D-1、D-2类操作耗时占比,对比B类操作(磁盘IO代价所在)和D类操作(CPU代价主要所在)耗时
回答实验目的2:分析D-1操作内部各子步骤耗时占比、D-2操作内部各子步骤耗时占比
实验结论
- 本次实验使用的人工数据集不具有代表性:其时间戳从1开始以步长1递增,值在[0,100)随机取整数,导致时间戳的编码压缩效率过高、值列的编码压缩效率过低。甚至对于值列来说,和其它编码方式相比,采用PLAIN编码的空间大小是最小的。
- 在中车ZT11529数据集上的实验结果来看:
- 相对其它压缩方法,GZIP的压缩率最高,但(B)磁盘加载IO代价和(D-1)解压缩步骤代价之间有tradeoff,再加上耗时测量的波动影响,导致GZIP压缩下的整体读耗时并不一定是最小的。
- 真实数据集的压缩率高,磁盘数据量相对小,此时从磁盘加载Chunk数据的操作耗时小于解压缩和解码Page数据的操作耗时,即整体耗时瓶颈不是磁盘IO。
- D-1步骤内部的主要耗时瓶颈是子步骤7_2_data_decompress_PageDataByteArray(us)。人工数据实验里发现另一个子步骤7_1_data_ByteBuffer_to_ByteArray(us)的占比也高,其主要因为人工数据实验里数据压缩率很低,从而子步骤7_2_data_decompress_PageDataByteArray(us)耗时少,从而相对来说7_1_data_ByteBuffer_to_ByteArray(us)占比高了。
- 本实验里,D-2类操作内部没有突出的耗时瓶颈子步骤。
实验设置
IoTDB版本
实验环境
- FIT楼166.111.130.101 / 192.168.130.31
- CPU:Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz,(6核12线程)
- L1 cache 284KB
- FIT楼166.111.130.101 / 192.168.130.31
- CPU:Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz,(6核12线程)
- L1 cache 284KB, L2 cache 1536KB, L3 cache 12MB
- 内存:16G
- 硬盘:1.8T HDD /dev/sdb1 mounted on /disk
- 操作系统:Ubuntu 16.04.7 LTS
- 工作文件夹:/disk/rl/tsfileReadExp/
数据集
...
(1)人工数据:时间戳从1开始以步长1递增,值在[0,100)随机取整数。
(2)中车数据:/disk/rl/zc_data
...
/ZT11529.csv,这条传感器时间序列数据共12,780,287个点,其部分数据作图和csv部分截图如下所示。
Image Added
Image Added
...
实验工具
...
WRITE_REAL
:用来标识是“写人工数据/写真实数据/读数据”中的“读数据”
path_of_tsfile_to_read
:要读取的TsFile地址
decomposeMeasureTime
:FALSE
to measure the read process as a whole, in which case D_decompose_each_step
is useless. TRUE
to measure the decomposed read process, and the decomposition granularity is controlled by D_decompose_each_step
.
D_decompose_each_step
:When decomposeMeasureTime
is TRUE
, D_decompose_each_step=FALSE
to measure the "(D_1)decompress_pageData" and "(D_2)decode_pageData" steps without further deomposition, D_decompose_each_step=TRUE
to break down these two steps further and measure substeps inside.
timeEncoding(te)
:If timeEncoding
is not specified, TS_2DIFF will be used by default. timeEncoding
should be the same with that used to write the TsFile.
控制参数 | decomposeMeasureTime=FALSE | decomposeMeasureTime=TRUE & D_decompose_each_step=FALSE 用于分析A/B/C/D-1/D-2类操作耗时占比(&实验目标1) | decomposeMeasureTime=TRUE & D_decompose_each_step=TRUE |
---|
测量最小单元步骤用于分析D-1操作内部各子步骤耗时占比、D-2操作内部各子步骤耗时占比(&实验目标2) |
---|
测量的最小单元步骤 | total_time(us) | - (A)1_index_read_deserialize_MagicString_FileMetadataSize(us)
- (A)2_index_read_deserialize_IndexRootNode_MetaOffset_BloomFilter(us)
- (A)3_2_index_read_deserialize_IndexRootNode_exclude_to_TimeseriesMetadata_forExactGet(us)
- (B)4_data_read_deserialize_ChunkHeader(us)
- (B)5_data_read_ChunkData(us)
- (C)6_data_deserialize_PageHeader(us)
- (D-1)7_data_decompress_PageData(us)
- (D-2)8_data_decode_PageData(us)
| - (A)1_index_read_deserialize_MagicString_FileMetadataSize(us)
- (A)2_index_read_deserialize_IndexRootNode_MetaOffset_BloomFilter(us)
- (A)3_2_index_read_deserialize_IndexRootNode_exclude_to_TimeseriesMetadata_forExactGet(us)
- (B)4_data_read_deserialize_ChunkHeader(us)
- (B)5_data_read_ChunkData(us)
- (C)6_data_deserialize_PageHeader(us)
- (D-1)7_1_data_ByteBuffer_to_ByteArray(us)
- (D-1)7_2_data_decompress_PageDataByteArray(us)
- (D-1)7_3_data_ByteArray_to_ByteBuffer(us)
- (D-1)7_4_data_split_time_value_Buffer(us)
- (D-2)8_1_createBatchData(us)
- (D-2)8_2_timeDecoder_hasNext(us)
- (D-2)8_3_timeDecoder_readLong(us)
- (D-2)8_4_valueDecoder_read(us)
- (D-2)8_5_checkValueSatisfyOrNot(us)
- (D-2
|
)8_6_putIntoBatchData(us)
Image Removed
Image Removed
结合实验目的,
...
- )8_6_putIntoBatchData(us)
|
自动化脚本
(1) RLUnitSynExp.sh:用人工数据写TsFile,清空系统缓存,然后进行一次读TsFile实验。
...
输入:
输出:不同压缩方式下的一个TsFile文件、一个TsFile空间统计结果文件( *writeResult.csv
)、REPEAT个读TsFile耗时结果csv文件( *readResult-T*csv
)、一个把重复读实验结果横向拼接起来的csv文件(*readResult-combined.csv
)、一个把写结果和读结果拼接起来的csv文件(*allResult-combined.csv
)、一个把读结果取平均值并且按照不同粒度统计百分比的csv文件( *allResult-combined-processed.csv
)
其它类似,不再赘述。
二、实验结果
改变压缩方式
人工数据实验结果
RLCompressionSynExpScripts.sh
其它类似,不再赘述。
实验结果
中车数据实验结果
改变压缩方式
RLCompressionRealExpScripts.sh
压缩方式 | GZIP | LZ4 | SNAPPY | UNCOMPRESSED |
dataset | synthetic | synthetic | synthetic | synthetic |
压缩方式 | GZIP | LZ4 | SNAPPY | UNCOMPRESSED |
dataset | /disk/rl/zc_data/ZT11529.csv | /disk/rl/zc_data/ZT11529.csv | /disk/rl/zc_data/ZT11529.csv | /disk/rl/zc_data/ZT11529.csv |
pagePointNum(ppn) | 10000 | 10000 | 10000 | 10000 |
numOfPagesInChunk(pic) |
10001000100010001010101013 |
timeEncoding(te) | TS_2DIFF | TS_2DIFF | TS_2DIFF | TS_2DIFF |
valueDataType(vt) |
INT64INT64INT64INT64PLAINPLAINPLAINPLAINGORILLA |
compression(co) | GZIP | LZ4 | SNAPPY | UNCOMPRESSED |
totalPointNum |
100000000100000000100000000100000000767131286677084443197679423904781422615130773735 |
chunkDataSize_stats_mean(MB) |
767130076177084364367679412647814216614837062438 |
compressedPageSize_stats_mean(B) |
8037541867807648144480460477898187429684.7575 |
uncompressedPageSize_stats_mean(B) |
81874 | 81874 | 81874 | 29684.7575 | 29684.7575 | 29684.7575 | 29684.7575 |
81874 |
timeBufferSize_stats_mean(B) |
1872 | 1872 | 1872 | | 11461.4625 | 11461.4625 | 11461.4625 | 11461.4625 |
1872 |
valueBufferSize_stats_mean(B) |
80000 | 80000 | 80000 | 18221.26833 | 18221.26833 | 18221.26833 | 18221.26833 |
80000 |
[2] category: (A)get ChunkStatistic->(B)load on-disk Chunk->(C)get PageStatistics->(D)load in-memory PageData |
[Avg&Per] (A)get |
_chunkMetadatas101654.7566 us - 0.9318259204986696% | 86735.1136 us - 1.1689451651044902% | 82918.49919999999 us - 1.0656107165107132% | _chunkMetadatas | 87316.85010000001 us - 8.19883087518919% | 100289.6416 us - 12.59103802007131% | 89466.6045 us - 11.199530364576052% | 88760.777 us - 10.335699109318087% |
[Avg&Per] (B)load_on_disk_chunk | 160699.6025 us - 15.089285299443361% | 176784.88239999997 us - 22.19476647997349% | 105802.9384 us - 13.244531050378352% | 191518.85280000002 us - 22.301305860612082% |
93896.1409 us - 1.2531889263513831%Bload_on_disk_chunk5949593.8917000005 us - 54.53739687304131% | 4849630.197000001 us - 65.35936296194448% | 5086618.179 us - 65.36966894765757% | 4851386.484999999 us - 64.74924062031137%get_pageHeader | 2436.5129999999995 us - 0.22878239411203669% | 2198.3668000000007 us - 0.27599779517870476% | 2319.9517000000005 us - 0.29041416798711567% | 3134.228800000001 us - 0.3649635223062446% |
[Avg&Per] ( |
CgetpageHeader6613381399999991 0060622054656159316%7120158900000014 009595969816001626%7692346500000014 009885667184764595%7605696300000007 010150975630087579%4515277590088287% |
[Avg&Per] (D_ |
1decompress2859428.0031000106 us - 26.211160404158996% | 521804.2723000004 us - 7.032452670194604% | 605202.8143000009 us - 7.777644443672276% | | 457950.96670000016 us - 43.000434862259155% | 486085.0215000002 us - 61.02639204858503% | 485623.25309999986 us - 60.790866035802786% | 545723.8052999998 us - 63.54650374875476% |
SUM | 1064991.4778 us | 796516.0731000002 us | 798842.4655999999 us | 858778.6472999998 us |
[3] D_1 compare each step inside |
498170.42259999976 us - 6.648853201570805%_2)decode_pageData1991910.319299996 us - 18.25899474764487% | 1954657.4198999994 us - 26.343279504596413% | 1998880.5966999987 us - 25.68821922031179% | 2041517.9070999995 us - 27.247207495465567% | -1)7_1_data_ByteBuffer_to_ByteArray(us) | 1269.7615999999996 us - 0.36377892330196115% | 1808.5776999999991 us - 7.207505165727991% | 1687.9916000000003 us - 1.412377327892232% | 3197.6313 us - 41.86667970732365% |
[3] D_1 compare each step inside1ByteBufferto_ByteArray6595237819999998 25136549346918415%10880934350000018 596896105506002%10813235939999981 43622622294156905%11076511740000003 63813731447511664%84820675254647% |
[Avg&Per] (D-1)7_ |
2decompressPageDataByteArray2554687926599999 9736728361519128%6890438600000006 3779892271446546%13534591170000008 54601079805416944%57547215800000035 3315396273496119%51469221169019% |
[Avg&Per] (D-1)7_ |
3ByteArraytoByteBuffer8118335000000624 03094155721322126%1239949800000022 0680200047933345%11847460000000272 047794876167766753%12291439000000355 0.7081314098345313%20.770421328439692% |
[3] D_2 compare each step inside |
[Avg&Per] (D- |
174_data_split_time_value_Buffer23120582000000422 08811989290364733%33382513999999974 18312666870009688%32183657999999955 129834913874849%4034201500000018 2.3241744076926314%[3] D_2 compare each step inside0.24721094086707432% |
[Avg&Per] (D-2)8_ |
1createBatchData53847852 0053292019060348375%58487599 005759123169122766%59134963 0058362326692940975%60193023 005943520403215091%537217392727715% |
[Avg&Per] (D-2)8_ |
2hasNext18598422956 18406444711361424%18622347849 18336946086748988%18640923926 18397368271414525%18577786739 18343895858133802%058508247673558% |
[Avg&Per] (D-2)8_ |
3timeDecoderreadLong20747577936 20533415498567617%20847004377 20527508047369906%20630438916 20360888969091857%20699304964 2043870456313607%5754416723178% |
[Avg&Per] (D-2)8_ |
4valueDecoder_readcheckValueSatisfyOrNot(us) |
1876012952 1856648209392724%18814715433999998 18526365490982297%18778092412 18532744562964893%18768431276 1853214021585961%979528915812129% |
[Avg&Per] (D-2)8_ |
5checkValueSatisfyOrNot17803796374 17620020492363725%17807823133 17534904586680103%17819492049 17586668929952697%17805995789 17.5818216126948%[Avg&Per] (D-2)8_6_putIntoBatchData(us) | 2507922.0072 us - 24.82034518471963% | 2540605.1784 us - 25.016684556527476% | 2539577.912 us - 25.063966939883077% | 2536332.2055 us - 25.044002546143567% |
B类操作耗时超过D类操作耗时。分析原因:使用的人工数据数值是INT64类型的随机取整的数,且PLAIN编码,且四种压缩方式此时的压缩率都不高,所以磁盘数据量偏大。
中车数据实验结果
RLCompressionRealExpScripts.sh
ZT11529传感器数据如下图所示,共12,780,287个点。
Image Removed
- 相对其它压缩方法,GZIP的压缩率最高,同时它的D-1解压缩步骤耗时占比也更高。
- 真实数据集的压缩率高,磁盘数据量少,D类操作耗时超过B类操作耗时,即整体耗时瓶颈在D类操作。
- 真实数据集的压缩率高,D-1步骤内部的主要耗时瓶颈是子步骤7_2_data_decompress_PageDataByteArray(us)。
- 人工数据实验里发现另一个子步骤7_1_data_ByteBuffer_to_ByteArray(us)的占比也高,其主要因为人工数据实验里数据压缩率很低,从而子步骤7_2_data_decompress_PageDataByteArray(us)耗时少,从而相对来说7_1_data_ByteBuffer_to_ByteArray(us)占比高了。
- 本实验里,D-2类操作内部没有突出的耗时瓶颈子步骤。
后续:可以增大真实数据集的数据量之后再实验看看。
改变编码方式
RLValueEncodingRealExpScripts.sh
编码方式 | GORILLA | PLAIN | RLE | TS_2DIFF |
压缩方式 | GZIP | LZ4 | SNAPPY | UNCOMPRESSED |
dataset | /disk/rl/zc_data/ZT11529.csv | /disk/rl/zc_data/ZT11529.csv | /disk/rl/zc_data/ZT11529.csv | /disk/rl/zc_data/ZT11529.csv |
pagePointNum(ppn) | 10000 | 10000 | 10000 | 10000 |
numOfPagesInChunk(pic) | 100 | 100 | 100 | 100 |
chunksWritten(cw) | 13 | 13 | 13 | 13 |
timeEncoding(te) | TS_2DIFF | TS_2DIFF | TS_2DIFF | TS_2DIFF |
valueDataType(vt) | DOUBLE | DOUBLE | DOUBLE | DOUBLE |
valueEncoding(ve) | GORILLA |
GORILLAGORILLAGORILLAGZIPLZ4UNCOMPRESSEDSNAPPY |
totalPointNum | 12780287 | 12780287 | 12780287 | 12780287 |
tsfileSize(MB) |
193486261423777410512315641212363077373598550797 |
chunkDataSize_stats_mean(MB) | 1. |
5151396591860070078131843412837062438642661015 |
compressedPageSize_stats_mean(B) |
15824166671944027518948692968475758925 |
uncompressedPageSize_stats_mean(B) | 29684.7575 |
29684757529684757529684757536417 |
timeBufferSize_stats_mean(B) | 11461.4625 | 11461.4625 | 11461.4625 | 11461.4625 |
valueBufferSize_stats_mean(B) | 18221.26833 |
18221.2683318221268331822126833875 |
[2] category: (A)get ChunkStatistic->(B)load on-disk Chunk->(C)get PageStatistics->(D)load |
in-memory PageData[Avg&Per] (A)get_chunkMetadatas | 87316.85010000001 us - 8.19883087518919% | 100289.6416 us - 12.59103802007131% | 89466.6045 us - 11.199530364576052% | 88760.777 us - 10.335699109318087%in-memory PageData |
[Avg&Per] ( |
Bload_on_disk_chunk160699.6025 us - 15.089285299443361% | 176784.88239999997 us - 22.19476647997349% | 105802.9384 us - 13.244531050378352% | get_chunkMetadatas | 88753.03020000001 us - 11.21274000219672% | 97256.88870000001 us - 12.181751942467722% | 84509.5108 us - 9.498496964807579% | 86370.7961 us - 11.6687579647861% |
191518.85280000002 us - 22.301305860612082%C)get_pageHeader2436.5129999999995 us - 0.22878239411203669% | 2198.3668000000007 us - 0.27599779517870476% | 2319.9517000000005 us - 0.29041416798711567% | B)load_on_disk_chunk | 105215.9241 us - 13.29260306229143% | 138228.6108 us - 17.313597737139155% | 139816.5536 us - 15.714765088895252% | 96826.5282 us - 13.081335047881257% |
3134.228800000001 us - 0.3649635223062446%D_1decompresspageData3565875454999999 33482666568996265%31158160799999983 3911805656191469%1156297179 14474658381255693%29640983400000016 34515277590088287%4268177375109232% |
[Avg&Per] (D_ |
2decode45795096670000016 43000434862259155%4860850215000002 6102639204858503%48562325309999986 60790866035802786%5457238052999998 63.54650374875476%[3] D_1 compare each step inside16.304565026949902% |
[Avg&Per] (D |
-17_1_data_ByteBuffer_to_ByteArray(us)decode_pageData | 480759.0216999998 us - 60.73737315655672% | 377408.39589999994 us - 47.27166909523449% | 506978.10549999995 us - 56.982107111140046% | 433147.3443000001 us - 58.51852422287182% |
SUM | 791537.3956999998 us | 798381.7858000002 us | 889714.5634 us | 740188.4276000002 |
[3] D_1 compare each step inside |
1269.7615999999996 us - 0.36377892330196115% | 1808.5776999999991 us - 7.207505165727991% | 1687.9916000000003 us - 1.412377327892232% | 3197.6313 us - 41.86667970732365%2decompressPageDataByteArray34585623130000004 9908569249502276%2124739440000001 8467466169480036%11610069619999993 9714384305311928%24324552000000012 3184820675254647%758246074195737% |
[Avg&Per] (D-1)7_ |
3ByteArrayto_ByteBuffer37418470000000025 010720162531460037%44210720000000003 1761876157051776%42426030000000026 035498732863644405%4211930000000002 551469221169019%4016073295714% |
[Avg&Per] (D-1)7_ |
4splittimevalue_Buffer15474220999999993 443326956360674%15948989000000004 6355956982419887%13012614999999998 10887922903520593%1586372500000001 20.770421328439692%[3] D_2 compare each step inside0.7256599483773118% |
[Avg&Per] (D- |
2)8_1_createBatchData1)7_4_data_split_time_value_Buffer(us) |
34308672 022855030377902266%41748958 275155541260775%34381465 022874736939899784%37306205 0.24721094086707432%2.1144866478555286% |
[3] D_2 compare each step inside |
[Avg&Per] (D-2)8_ |
2timeDecoder_hasNext23401677980000002 15589238228946503%2361359462 1556305048087338%2352784135 1565358490817499%2344696457 15537217392727715%22194982158403212% |
[Avg&Per] (D-2)8_ |
3readLong3578939434 23841426880277478%3602531425 23743262865502565%3583411524 2384121675993312%3630630738 24058508247673558%969786637750962% |
[Avg&Per] (D-2)8_ |
4valueDecoderread3538216809 570149451806063%3594778899 23692168165422427%3564404841 714761161335133%3557731939 5754416723178%788389623148674% |
[Avg&Per] (D-2)8_ |
5checkValueSatisfyOrNot2237583096 14905861011513531%2249382721 14825043539994219%2242821638 14921980483485841%2260536587 14979528915812129%298501890735768% |
[Avg&Per] (D-2)8_ |
6putIntoBatchDatacheckValueSatisfyOrNot(us) |
3282215562 21864774123677403%332305597 2190131940694663%3252517878 21639709317671908%3259937043 21602092830601723%...
312273986066641% |
[Avg&Per] (D-2)8_6_putIntoBatchData(us) |
...
- 人工数据实验里发现另一个子步骤7_1_data_ByteBuffer_to_ByteArray(us)的占比也高,其主要因为人工数据实验里数据压缩率很低,从而子步骤7_2_data_decompress_PageDataByteArray(us)耗时少,从而相对来说7_1_data_ByteBuffer_to_ByteArray(us)占比高了。
| 327591.9399 us - 21.699128552797042% | 329856.9807 us - 23.30331261469754% | 330303.223 us - 21.225373315680617% | 337085.3345 us - 22.409098040713936% |
人工数据实验结果
改变压缩方式
RLCompressionSynExpScripts.sh
压缩方式 | GZIP | LZ4 | SNAPPY | UNCOMPRESSED |
...
后续:可以增大真实数据集的数据量之后再实验看看。
改变编码方式
人工数据实验结果
RLValueEncodingSynExpScripts.sh
编码方式 | GORILLA | PLAIN | RLE | TS_2DIFF |
dataset | synthetic | synthetic | synthetic | synthetic |
pagePointNum(ppn) | 10000 | 10000 | 10000 | 10000 |
numOfPagesInChunk(pic) | 1000 | 1000 | 1000 | 1000 |
chunksWritten(cw) | 10 | 10 | 10 | 10 |
timeEncoding(te) | TS_2DIFF | TS_2DIFF | TS_2DIFF | TS_2DIFF |
valueDataType(vt) | INT64 | INT64 | INT64 | INT64 |
valueEncoding(ve) |
GORILLARLETS_2DIFFUNCOMPRESSEDUNCOMPRESSEDUNCOMPRESSEDSNAPPY | UNCOMPRESSED |
totalPointNum | 100000000 | 100000000 | 100000000 | 100000000 |
tsfileSize(MB) |
80538128957814226151781842231879332447054226151 |
chunkDataSize_stats_mean(MB) |
805380362478142166147818412781793323516814216614 |
compressedPageSize_stats_mean( |
B)84386.25189 | 81874 | 81918 | 83122 | B) | 80375.41867 | 80764.81444 | 80460.47789 | 81874 |
uncompressedPageSize_stats_mean(B) |
84386.25189819188312281874 |
timeBufferSize_stats_mean(B) | 1872 | 1872 | 1872 | 1872 |
valueBufferSize_stats_mean(B) |
82512.25189800448124880000 |
[2] category: (A)get ChunkStatistic->(B)load on-disk Chunk->(C)get PageStatistics->(D)load in-memory PageData |
[Avg&Per] (A)get_chunkMetadatas |
9133198490000001 8580518676474486%1009447581 2939556377951902%8809820449999999 09671805828234409%882312157 09257222461823116%2531889263513831% |
[Avg&Per] (B)load_on_disk_chunk |
5552645935400001 5216637107440095%51701583812 6627343223726832%5270914364100001 57866400973957255%55260996186 5797985793317826%74924062031137% |
[Avg&Per] (C)get_pageHeader |
8185805399999992 07690455451459878%7712402700000001 09886107156476356%781390999999998 08578451820695045%7725986500000003 08106107934716146%10150975630087579% |
[Avg&Per] (D_1)decompress_pageData |
5481603352000009 5149893543905802%5254410348000005 67353412114264035%5850362351000007 6422783415941812%6321546457000006 632568914631739%648853201570805% |
[Avg&Per] (D_2)decode_pageData |
4443785968300003 41748778959531215%1996996816400002 25598409841945312%3156902088299994 3465785050907054%3276856417400001 3438078982666053%247207495465567% |
[3] D_1 compare each step inside |
[Avg&Per] (D-1)7_1_data_ByteBuffer_to_ByteArray(us) |
11042172189999989 6493156378407389%10968791739999971 6392643734398307%11365835759999987 62901942587101885%10918747799999996 08420663361515%813731447511664% |
[Avg&Per] (D-1)7_2_data_decompress_PageDataByteArray(us) |
5462426410000002 3212084386602246%57095555900000094 3327545607007074%62164880499999985 34403908579311135%5907225890000005 3412961499817788%15396273496119% |
[Avg&Per] (D-1)7_3_data_ByteArray_to_ByteBuffer(us) |
11793427000000347 6934918640164246%123410400000004 7192394012210651%1032115900000022 5712038820191121%11932719000000652 6894253122110825%7081314098345313% |
[Avg&Per] (D-1)7_4_data_split_time_value_Buffer(us) |
38332921999999994 2254100485887217%35670158000000133 20788671847251163%38359775000000177 2122944951567873%36291045000000054 0967530559958814%3241744076926314% |
[3] D_2 compare each step inside |
[Avg&Per] (D-2)8_1_createBatchData(us) |
60089294 04720551599260821%6005094 058953749593284185%91361988 07959246314858166%6219250599999999 053213386787106826%05943520403215091% |
[Avg&Per] (D-2)8_2_timeDecoder_hasNext(us) |
17950674479 1410186066084482%18626318749 285997377780266%18051002815999999 15725618584694368%18386618702 15732028111177547%343895858133802% |
[Avg&Per] (D-2)8_3_timeDecoder_readLong(us) |
20736152138 16290102549307967%20894938765 513167449482335%20634693846 17976470800088325%21720295514 1858440125112089%43870456313607% |
[Avg&Per] (D-2)8_4_valueDecoder_read(us) |
46361954124 3642146247963989%1880348902 459930571697104%33524949242 29206164899804453%33645582458 28787960289219587%53214021585961% |
[Avg&Per] (D-2)8_5_checkValueSatisfyOrNot(us) |
1724239435 13545443257159627%17842052864 516060810611705%1723128802 1501150190311584%17808073196 15237010821076382%5818216126948% |
[Avg&Per] (D-2)8_6_putIntoBatchData(us) |
24941685891 1959392553705509%25634253348 165890040835308%25253939444 22000651349148434%25251035280999998 21605386140618513%...
B类操作耗时超过D类操作耗时。分析原因:使用的人工数据数值是INT64类型的随机取整的数,且PLAIN编码,且四种压缩方式此时的压缩率都不高,所以磁盘数据量偏大。
...
改变编码方式
RLValueEncodingRealExpScriptsRLValueEncodingSynExpScripts.sh
ZT11529传感器数据如下图所示,共12,780,287个点。
...
编码方式GORILLAPLAINRLETS_2DIFF | GORILLA | PLAIN | RLE | TS_2DIFF |
dataset | /disk/rl/zc_data/ZT11529.csv | /disk/rl/zc_data/ZT11529.csv | /disk/rl/zc_data/ZT11529.csv | synthetic | synthetic | synthetic | synthetic/disk/rl/zc_data/ZT11529.csv |
pagePointNum(ppn) | 10000 | 10000 | 10000 | 10000 |
numOfPagesInChunk(pic) | 1001000100 | 1000 | 1001000100 | 1000 |
chunksWritten(cw) | 131013 | 10 | 131013 | 10 |
timeEncoding(te) | TS_2DIFF | TS_2DIFF | TS_2DIFF | TS_2DIFF |
valueDataType(vt) | DOUBLEINT64DOUBLE | INT64 | DOUBLEINT64DOUBLE | INT64 |
valueEncoding(ve) | GORILLA | PLAIN | RLE | TS_2DIFF |
compression(co) | SNAPPYUNCOMPRESSEDSNAPPY | UNCOMPRESSED | SNAPPYUNCOMPRESSEDSNAPPY | UNCOMPRESSED |
totalPointNum | 1278028710000000012780287 | 100000000 | 1278028710000000012780287 | 100000000 |
tsfileSize(MB) | 23805.156412123812895 | 26781.599601754226151 | 22781.101478588422318 | 20793.985507973244705 |
chunkDataSize_stats_mean(MB) | 180.81318434153803624 | 278.08116674414216614 | 178.740072568184127811 | 79.64266101533235168 |
compressedPageSize_stats_mean(B) | 1894884386.6925189 | 21758.61667 | 18182.29917 | 81874 | 81918 | 8312217160.8925 |
uncompressedPageSize_stats_mean(B) | 2968484386.757525189 | 91463.48917 | 25245.93417 | 81874 | 81918 | 8312222210.36417 |
timeBufferSize_stats_mean(B) | 11461.4625 | 11461.4625 | 11461.4625 | 1872 | 1872 | 1872 | 187211461.4625 |
valueBufferSize_stats_mean(B) | 1822182512.2683325189 | 80000 | 13782.445 | 80044 | 8124810746.875 |
[2] category: (A)get ChunkStatistic->(B)load on-disk Chunk->(C)get PageStatistics->(D)load in-memory PageData |
[Avg&Per] (A)get_chunkMetadatas | 8875391331.03020000001 98490000001 us - 110.21274000219672%8580518676474486% | 97256100944.88870000001 7581 us - 121.181751942467722%2939556377951902% | 8450988098.5108 20449999999 us - 90.498496964807579%9671805828234409% | 8637088231.7961 2157 us - 110.6687579647861%9257222461823116% |
[Avg&Per] (B)load_on_disk_chunk | 1052155552645.9241 935400001 us - 1352.29260306229143%16637107440095% | 1382285170158.6108 3812 us - 1766.313597737139155%27343223726832% | 1398165270914.5536 364100001 us - 1557.714765088895252%866400973957255% | 968265526099.5282 6186 us - 1357.081335047881257%97985793317826% |
[Avg&Per] (C)get_pageHeader | 23528185.1583 805399999992 us - 0.2971632563133493%07690455451459878% | 26257712.5312999999983 402700000001 us - 0.328856613051254%09886107156476356% | 33887813.6254999999987 90999999998 us - 0.38086658793698225%08578451820695045% | 31597725.255499999999 986500000003 us - 0.4268177375109232%08106107934716146% |
[Avg&Per] (D_1)decompress_pageData | 114457548160.26139999992 3352000009 us - 145.460120522641777%149893543905802% | 182862525441.35910000015 0348000005 us - 226.904124612107367%7353412114264035% | 155021585036.76800000004 2351000007 us - 176.42376424722015%422783415941812% | 120684632154.50350000002 6457000006 us - 166.304565026949902%632568914631739% |
[Avg&Per] (D_2)decode_pageData | 4807594443785.0216999998 968300003 us - 6041.73737315655672%748778959531215% | 3774081996996.39589999994 816400002 us - 4725.27166909523449%598409841945312% | 5069783156902.10549999995 088299994 us - 5634.982107111140046%65785050907054% | 4331473276856.3443000001 417400001 us - 5834.51852422287182%38078982666053% |
[3] D_1 compare each step inside |
[Avg&Per] (D-1)7_1_data_ByteBuffer_to_ByteArray(us) | 1754110421.3673999999999 72189999989 us - 164.6860113426598728%93156378407389%2320 | 109687.3632000000002 91739999971 us - 163.489795195953362%92643734398307%1825 | 113658.1960999999988 35759999987 us - 162.548613686329812%901942587101885%1607 | 109187.3271999999997 47799999996 us - 163.758246074195737%08420663361515% |
[Avg&Per] (D-1)7_2_data_decompress_PageDataByteArray(us) | 10052754624.56879999992 26410000002 us - 9632.61067644486589%12084386602246% | 15153257095.73299999998 555900000094 us - 9733.29198327791244%27545607007074% | 11368662164.73430000001 880499999985 us - 9634.4591326329927%403908579311135% | 8721259072.82 25890000005 us - 9534.4016073295714%12961499817788% |
[Avg&Per] (D-1)7_3_data_ByteArray_to_ByteBuffer(us) | 4311179.9958999999999 3427000000347 us - 0.41516388607230165%6934918640164246% | 4681234.2119000000003 10400000004 us - 0.30061666178303303%7192394012210651% | 4741032.28859999999975 115900000022 us - 0.4024169332984032%5712038820191121% | 6631193.3730000000002 2719000000652 us - 0.7256599483773118%6894253122110825% |
[Avg&Per] (D-1)7_4_data_split_time_value_Buffer(us) | 13403833.373799999999 2921999999994 us - 12.288148326401935%254100485887217% | 14293567.1740000000002 0158000000133 us - 02.9176048643511714%0788671847251163% | 18733835.7816000000007 9775000000177 us - 12.5898367473790769%122944951567873% | 19323629.989900000001 1045000000054 us - 2.1144866478555286%0967530559958814% |
[3] D_2 compare each step inside |
[Avg&Per] (D-2)8_1_createBatchData(us) | 33486008.9259 9294 us - 0.2218271110091239%04720551599260821% | 34426005.3069 094 us - 0.24318767981262365%058953749593284185% | 34669136.6887 1988 us - 0.22277034162258696%07959246314858166% | 33386219.6453 250599999999 us - 0.22194982158403212%053213386787106826% |
[Avg&Per] (D-2)8_2_timeDecoder_hasNext(us) | 2353171795067.741 4479 us - 1514.587043790733997%10186066084482% | 2448391862631.8323 8749 us - 1718.29713023052909%285997377780266% | 2393081805100.8755 2815999999 us - 15.37805224577913%725618584694368% | 2402231838661.005 8702 us - 15.969786637750962%732028111177547% |
[Avg&Per] (D-2)8_3_timeDecoder_readLong(us) | 3615622073615.5392 2138 us - 2316.949282819264262%290102549307967% | 3633642089493.4154 8765 us - 2520.67050285597614%513167449482335% | 3621432063469.5815 3846 us - 2317.271443255687238%976470800088325%357833 | 2172029.1113 5514 us - 2318.788389623148674%58440125112089% |
[Avg&Per] (D-2)8_4_valueDecoder_read(us) | 3563254636195.4946 4124 us - 2336.602389962111484%42146247963989% | 2416251880348.5526 902 us - 1718.07005192367858%459930571697104% | 3964383352494.5566 9242 us - 2529.475247513902037%206164899804453% | 3354213364558.7093 2458 us - 2228.298501890735768%787960289219587% |
[Avg&Per] (D-2)8_5_checkValueSatisfyOrNot(us) | 2255541724239.2633 435 us - 1413.940327764084076%545443257159627% | 2323641784205.86419999998 2864 us - 1617.415814695306036%516060810611705% | 2245101723128.63450000001 802 us - 1415.4271133273284%01150190311584% | 2303321780807.4743 3196 us - 15.312273986066641%237010821076382% |
[Avg&Per] (D-2)8_6_putIntoBatchData(us) | 3275912494168.9399 5891 us - 2119.699128552797042%59392553705509% | 3298562563425.9807 3348 us - 2325.30331261469754%165890040835308% | 3303032525393.223 9444 us - 2122.225373315680617%000651349148434%337085 | 2525103.3345 5280999998 us - 2221.409098040713936%605386140618513% |
- 总的来说,使用的人工数据集不太好,其随机生成的取值,编码压缩效率不高,甚至PLAIN编码的空间是最小的