Table of Contents |
---|
一、实验设置
实验目的
- 读一个TsFile的磁盘IO代价和CPU代价的大小比较
- 了解CPU代价里有没有突出的耗时瓶颈操作
IoTDB版本
- v0.13.1
实验环境
- CPU:Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz,(6核12线程)
- L1 cache 284KB, L2 cache 1536KB, L3 cache 12MB
- 内存:16G
- 硬盘:1.8T HDD /dev/sdb1 mounted on /disk
- 操作系统:Ubuntu 16.04.7 LTS
...
实验设计
分析TsFile文件结构,然后把读取TsFile的耗时步骤进行分解和分类,最后进行实验测量统计。
TsFile结构
TsFile读耗时步骤分解与分类
耗时分类 | 耗时分解 |
(A)get_chunkMetadatas | (A)1_index_read_deserialize_MagicString_FileMetadataSize |
(A)2_index_read_deserialize_IndexRootNode_MetaOffset_BloomFilter | |
(A)3_2_index_read_deserialize_IndexRootNode_exclude_to_TimeseriesMetadata_forExactGet | |
(B)load_on_disk_chunk | (B)4_data_read_deserialize_ChunkHeader |
(B)5_data_read_ChunkData | |
(C)get_pageHeader | (C)6_data_deserialize_PageHeader |
(D-1)decompress_pageData | (D-1)7_1_data_ByteBuffer_to_ByteArray |
(D-1)7_2_data_decompress_PageDataByteArray | |
(D-1)7_3_data_ByteArray_to_ByteBuffer | |
(D-1)7_4_data_split_time_value_Buffer | |
(D-2)decode_pageData | (D-2)8_1_createBatchData |
(D-2)8_2_timeDecoder_hasNext | |
(D-2)8_3_timeDecoder_readLong | |
(D-2)8_4_valueDecoder_read | |
(D-2)8_5_checkValueSatisfyOrNot | |
(D-2)8_6_putIntoBatchData |
回答实验目的1:分析A、B、C、D-1、D-2类操作耗时占比,对比B类操作(磁盘IO代价所在)和D类操作(CPU代价主要所在)耗时
回答实验目的2:分析D-1操作内部各子步骤耗时占比、D-2操作内部各子步骤耗时占比
实验结论
- 本次实验使用的人工数据集不具有代表性:其编码压缩效率过低。甚至对于值列来说,和其它编码方式相比,采用PLAIN编码的空间大小是最小的。
- 在中车ZT11529数据集上的实验结果来看:
- 真实数据集的压缩率高、磁盘数据量相对小,此时【从磁盘加载Chunk数据的耗时】小于【解压缩和解码Page数据的耗时】
- D-1步骤内部的耗时瓶颈就是子步骤7_2_data_decompress_PageDataByteArray。注意:人工数据实验里发现另一个子步骤7_1_data_ByteBuffer_to_ByteArray(us)的占比也高,分析是因为人工数据压缩率很低,子步骤7_2_data_decompress_PageDataByteArray(us)耗时相对少,从而7_1_data_ByteBuffer_to_ByteArray(us)耗时占比相对偏高。
- D-1解压缩和D-2解码的空间压缩关系和耗时关系还有待探索
- 写数据的耗时可以也测量一下
- 注意RLE编码对于浮点数是有损的
实验设置
IoTDB版本
- v0.13.1
实验环境
- CPU:Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz,(6核12线程)
- L1 cache 284KB, L2 cache 1536KB, L3 cache 12MB
- 内存:16G
- 硬盘:1.8T HDD /dev/sdb1 mounted on /disk
- 操作系统:Ubuntu 16.04.7 LTS
作图和csv部分截图如下所示。
RLTsFileReadCostBench用法
(1) 用人工数据写TsFile
Code Block | ||
---|---|---|
| ||
java -jar RLTsFileReadCostBench-0.13.1-jar-with-dependencies.jar WRITE_SYN [pagePointNum] [numOfPagesInChunk] [chunksWritten] [timeEncoding] [valueDataType] [valueEncoding] [compressionType] |
:用来标识是“写人工数据/写真实数据/读数据”中的“写人工数据”
pagePointNum(ppn)
:一个page内的点数numOfPagesInChunk(pic)
:一个chunk内的pages数chunksWritten(cw)
:写的chunks总数timeEncoding(te)
:时间戳列编码方式valueDataType(vt)
:值列数据类型valueEncoding(ve)
:值列编码方式compressionType(co)
Code Block | ||
---|---|---|
| ||
java -jar RLTsFileReadCostBench-0.13.1-jar-with-dependencies.jar WRITE_REAL [path_of_real_data_csv_to_write] [pagePointNum] [numOfPagesInChunk] [timeEncoding] [valueDataType] [valueEncoding] [compressionType] |
:用来标识是“写人工数据/写真实数据/读数据”中的“写真实数据”
path_of_real_data_csv_to_write
:用来写TsFile的真实数据集csv地址pagePointNum(ppn)
:一个page内的点数numOfPagesInChunk(pic)
:一个chunk内的pages数timeEncoding(te)
:时间戳列编码方式valueDataType(vt)
:值列数据类型valueEncoding(ve)
:值列编码方式compressionType(co)
(3) 读实验
Code Block | ||
---|---|---|
| ||
java -jar RLTsFileReadCostBench-0.13.1-jar-with-dependencies.jar READ [path_of_tsfile_to_read] [decomposeMeasureTime] [D_decompose_each_step] (timeEncoding) |
:用来标识是“写人工数据/写真实数据/读数据”中的“读数据”
path_of_tsfile_to_read
:要读取的TsFile地址decomposeMeasureTime
:FALSE
to measure the read process as a whole, in which caseD_decompose_each_step
is useless.TRUE
to measure the decomposed read process, and the decomposition granularity is controlled byD_decompose_each_step
.D_decompose_each_step
:WhendecomposeMeasureTime
isTRUE
,D_decompose_each_step=FALSE
to measure the "(D_1)decompress_pageData" and "(D_2)decode_pageData" steps without further deomposition,D_decompose_each_step=TRUE
to break down these two steps further and measure substeps inside.timeEncoding(te)
:IftimeEncoding
is not specified, TS_2DIFF will be used by default.timeEncoding
控制参数 | decomposeMeasureTime=FALSE | decomposeMeasureTime=TRUE & D_decompose_each_step=FALSE 用于分析A/B/C/D-1/D-2类操作耗时占比(&实验目标1) | |
---|---|---|---|
测量的最小单元步骤 | total_time(us) |
|
|
自动化脚本
- 输出:一个TsFile文件、一个TsFile空间统计结果文件(
*writeResult.csv
)、一个读TsFile耗时结果csv文件(*readResult-T*csv
输入:见RLTsFileReadCostBench的写数据参数和读数据参数
输出:一个TsFile文件、一个TsFile空间统计结果文件(
*writeResult.csv
)、一个读TsFile耗时结果csv文件(*readResult-T*csv
WRITE_READ_JAR_PATH
:RLTsFileReadCostBench-0.13.1-jar-with-dependencies.jar的地址Calculator_JAR_PATH
:把若干次重复读实验结果进行平均值和百分比计算的RLRepeatReadResultAvgPercCalculator-0.13.1-jar-with-dependencies.jar的地址FILE_NAME
:要读取的TsFile的地址decomposeMeasureTime
:见RLTsFileReadCostBench读数据参数D_decompose_each_step
:见RLTsFileReadCostBench读数据参数te
::见RLTsFileReadCostBench读数据参数REPEAT
:读实验重复次数
输出:
REPEAT个读TsFile耗时结果csv文件
*readResult-T*csv
一个把重复读实验结果横向拼接起来的csv文件
*readResult-combined.csv
一个把写结果和读结果拼接起来的csv文件
*allResult-combined.csv
一个把读结果取平均值并且按照不同粒度统计百分比的csv文件
工具地址:
WRITE_READ_JAR_PATH
:RLTsFileReadCostBench-0.13.1-jar-with-dependencies.jar的地址Calculator_JAR_PATH
:把若干次重复读实验结果进行平均值和百分比统计计算的RLRepeatReadResultAvgPercCalculator-0.13.1-jar-with-dependencies.jar的地址TOOL_PATH
:用于替换脚本中变量值的自动脚本工具RLtool.sh的地址READ_SCRIPT_PATH
:RLReadExpScripts.sh的地址
写数据参数:见RLTsFileReadCostBench写数据参数
读数据参数:见RLTsFileReadCostBench读数据参数
REPEAT
:读实验重复次数
输出:不同压缩方式下的一个TsFile文件、一个TsFile空间统计结果文件(
*writeResult.csv
)、REPEAT个读TsFile耗时结果csv文件(*readResult-T*csv
)、一个把重复读实验结果横向拼接起来的csv文件(*readResult-combined.csv
)、一个把写结果和读结果拼接起来的csv文件(*allResult-combined.csv
)、一个把读结果取平均值并且按照不同粒度统计百分比的csv文件(*allResult-combined-processed.csv
其它类似,不再赘述。
实验具体结果
中车数据实验结果
改变压缩方式
RLCompressionRealExpScripts.sh
压缩方式 | GZIP | LZ4 | SNAPPY | UNCOMPRESSED |
dataset | /disk/rl/zc_data/ZT11529.csv | /disk/rl/zc_data/ZT11529.csv | /disk/rl/zc_data/ZT11529.csv | /disk/rl/zc_data/ZT11529.csv |
pagePointNum(ppn) | 10000 | 10000 | 10000 | 10000 |
numOfPagesInChunk(pic) | 100 | 100 | 100 | 100 |
chunksWritten(cw) | 13 | 13 | 13 | 13 |
timeEncoding(te) | TS_2DIFF | TS_2DIFF | TS_2DIFF | TS_2DIFF |
valueDataType(vt) | DOUBLE | DOUBLE | DOUBLE | DOUBLE |
valueEncoding(ve) | GORILLA | GORILLA | GORILLA | GORILLA |
compression(co) | GZIP | LZ4 | SNAPPY | UNCOMPRESSED |
totalPointNum | 12780287 | 12780287 | 12780287 | 12780287 |
tsfileSize(MB) | 19.34862614 | 23.77741051 | 23.15641212 | 36.30773735 |
chunkDataSize_stats_mean(MB) | 1.515139659 | 1.86007007 | 1.813184341 | 2.837062438 |
compressedPageSize_stats_mean(B) | 15824.16667 | 19440.275 | 18948.69 | 29684.7575 |
uncompressedPageSize_stats_mean(B) | 29684.7575 | 29684.7575 | 29684.7575 | 29684.7575 |
timeBufferSize_stats_mean(B) | 11461.4625 | 11461.4625 | 11461.4625 | 11461.4625 |
valueBufferSize_stats_mean(B) | 18221.26833 | 18221.26833 | 18221.26833 | 18221.26833 |
[2] category: (A)get ChunkStatistic->(B)load on-disk Chunk->(C)get PageStatistics->(D)load in-memory PageData | ||||
[Avg&Per] (A)get_chunkMetadatas | 87316.85010000001 us - 8.19883087518919% | 100289.6416 us - 12.59103802007131% | 89466.6045 us - 11.199530364576052% | 88760.777 us - 10.335699109318087% |
[Avg&Per] (B)load_on_disk_chunk | 160699.6025 us - 15.089285299443361% | 176784.88239999997 us - 22.19476647997349% | 105802.9384 us - 13.244531050378352% | 191518.85280000002 us - 22.301305860612082% |
[Avg&Per] (C)get_pageHeader | 2436.5129999999995 us - 0.22878239411203669% | 2198.3668000000007 us - 0.27599779517870476% | 2319.9517000000005 us - 0.29041416798711567% | 3134.228800000001 us - 0.3649635223062446% |
[Avg&Per] (D_1)decompress_pageData | 356587.5454999999 us - 33.482666568996265% | 31158.160799999983 us - 3.911805656191469% | 115629.7179 us - 14.474658381255693% | 29640.983400000016 us - 3.4515277590088287% |
[Avg&Per] (D_2)decode_pageData | 457950.96670000016 us - 43.000434862259155% | 486085.0215000002 us - 61.02639204858503% | 485623.25309999986 us - 60.790866035802786% | 545723.8052999998 us - 63.54650374875476% |
SUM | 1064991.4778 us | 796516.0731000002 us | 798842.4655999999 us | 858778.6472999998 us |
[3] D_1 compare each step inside | ||||
[Avg&Per] (D-1)7_1_data_ByteBuffer_to_ByteArray(us) | 1269.7615999999996 us - 0.36377892330196115% | 1808.5776999999991 us - 7.207505165727991% | 1687.9916000000003 us - 1.412377327892232% | 3197.6313 us - 41.86667970732365% |
[Avg&Per] (D-1)7_2_data_decompress_PageDataByteArray(us) | 345856.23130000004 us - 99.08569249502276% | 21247.39440000001 us - 84.67466169480036% | 116100.69619999993 us - 97.14384305311928% | 2432.4552000000012 us - 31.84820675254647% |
[Avg&Per] (D-1)7_3_data_ByteArray_to_ByteBuffer(us) | 374.18470000000025 us - 0.10720162531460037% | 442.10720000000003 us - 1.761876157051776% | 424.26030000000026 us - 0.35498732863644405% | 421.1930000000002 us - 5.51469221169019% |
[Avg&Per] (D-1)7_4_data_split_time_value_Buffer(us) | 1547.4220999999993 us - 0.443326956360674% | 1594.8989000000004 us - 6.355956982419887% | 1301.2614999999998 us - 1.0887922903520593% | 1586.372500000001 us - 20.770421328439692% |
[3] D_2 compare each step inside | ||||
[Avg&Per] |
RLTsFileReadCostBench用法
(1) 用人工数据写TsFile
Code Block | ||
---|---|---|
| ||
java -jar RLTsFileReadCostBench-0.13.1-jar-with-dependencies.jar WRITE_SYN [pagePointNum] [numOfPagesInChunk] [chunksWritten] [timeEncoding] [valueDataType] [valueEncoding] [compressionType] |
:用来标识是“写人工数据/写真实数据/读数据”中的“写人工数据”
pagePointNum(ppn)
:一个page内的点数numOfPagesInChunk(pic)
:一个chunk内的pages数chunksWritten(cw)
:写的chunks总数timeEncoding(te)
:时间戳列编码方式valueDataType(vt)
:值列数据类型valueEncoding(ve)
:值列编码方式compressionType(co)
Code Block | ||
---|---|---|
| ||
java -jar RLTsFileReadCostBench-0.13.1-jar-with-dependencies.jar WRITE_REAL [path_of_real_data_csv_to_write] [pagePointNum] [numOfPagesInChunk] [timeEncoding] [valueDataType] [valueEncoding] [compressionType] |
:用来标识是“写人工数据/写真实数据/读数据”中的“写真实数据”
path_of_real_data_csv_to_write
:用来写TsFile的真实数据集csv地址pagePointNum(ppn)
:一个page内的点数numOfPagesInChunk(pic)
:一个chunk内的pages数timeEncoding(te)
:时间戳列编码方式valueDataType(vt)
:值列数据类型valueEncoding(ve)
:值列编码方式compressionType(co)
(3) 读实验
Code Block | ||
---|---|---|
| ||
java -jar RLTsFileReadCostBench-0.13.1-jar-with-dependencies.jar READ [path_of_tsfile_to_read] [decomposeMeasureTime] [D_decompose_each_step] (timeEncoding) |
:用来标识是“写人工数据/写真实数据/读数据”中的“读数据”
path_of_tsfile_to_read
:要读取的TsFile地址decomposeMeasureTime
:FALSE
to measure the read process as a whole, in which caseD_decompose_each_step
is useless.TRUE
to measure the decomposed read process, and the decomposition granularity is controlled byD_decompose_each_step
.D_decompose_each_step
:WhendecomposeMeasureTime
isTRUE
,D_decompose_each_step=FALSE
to measure the "(D_1)decompress_pageData" and "(D_2)decode_pageData" steps without further deomposition,D_decompose_each_step=TRUE
to break down these two steps further and measure substeps inside.timeEncoding(te)
:IftimeEncoding
is not specified, TS_2DIFF will be used by default.timeEncoding
控制参数 | decomposeMeasureTime=FALSE | decomposeMeasureTime=TRUE & D_decompose_each_step=FALSE | ||||
---|---|---|---|---|---|---|
测量最小单元步骤 | total_time(us) | 3430.8672 us - 0.22855030377902266% | 4174.8958 us - 0.275155541260775% | 3438.1465 us - 0.22874736939899784% | 3730.6205 us - 0.24721094086707432% | |
[Avg&Per] (D-2)8_2_timeDecoder_hasNext(us)(D-2)8_3_timeDecoder_readLong(us) (D-2)8_4_valueDecoder_read(us)(D-2)8_5timeDecoder_checkValueSatisfyOrNothasNext(us) |
结合实验目的,
- 目的1:decomposeMeasureTime=TRUE & D_decompose_each_step=FALSE:对比B类操作(磁盘IO代价所在)和D类操作(CPU代价主要所在)耗时
- 目的2:decomposeMeasureTime=TRUE & D_decompose_each_step=TRUE:分析D-1操作内部各子步骤耗时占比,分析D-2操作内部各子步骤耗时占比
自动化脚本
- 输出:一个TsFile文件、一个TsFile空间统计结果文件(
*writeResult.csv
)、一个读TsFile耗时结果csv文件(*readResult-T*csv
输入:见RLTsFileReadCostBench的写数据参数和读数据参数
输出:一个TsFile文件、一个TsFile空间统计结果文件(
*writeResult.csv
)、一个读TsFile耗时结果csv文件(*readResult-T*csv
WRITE_READ_JAR_PATH
:RLTsFileReadCostBench-0.13.1-jar-with-dependencies.jar的地址Calculator_JAR_PATH
:把若干次重复读实验结果进行平均值和百分比计算的RLRepeatReadResultAvgPercCalculator-0.13.1-jar-with-dependencies.jar的地址FILE_NAME
:要读取的TsFile的地址decomposeMeasureTime
:见RLTsFileReadCostBench读数据参数D_decompose_each_step
:见RLTsFileReadCostBench读数据参数te
::见RLTsFileReadCostBench读数据参数REPEAT
:读实验重复次数
输出:
REPEAT个读TsFile耗时结果csv文件
*readResult-T*csv
一个把重复读实验结果横向拼接起来的csv文件
*readResult-combined.csv
一个把写结果和读结果拼接起来的csv文件
*allResult-combined.csv
一个把读结果取平均值并且按照不同粒度统计百分比的csv文件
工具地址:
WRITE_READ_JAR_PATH
:RLTsFileReadCostBench-0.13.1-jar-with-dependencies.jar的地址Calculator_JAR_PATH
:把若干次重复读实验结果进行平均值和百分比统计计算的RLRepeatReadResultAvgPercCalculator-0.13.1-jar-with-dependencies.jar的地址TOOL_PATH
:用于替换脚本中变量值的自动脚本工具RLtool.sh的地址READ_SCRIPT_PATH
:RLReadExpScripts.sh的地址
写数据参数:见RLTsFileReadCostBench写数据参数
读数据参数:见RLTsFileReadCostBench读数据参数
REPEAT
:读实验重复次数
输出:不同压缩方式下的一个TsFile文件、一个TsFile空间统计结果文件(
*writeResult.csv
)、REPEAT个读TsFile耗时结果csv文件(*readResult-T*csv
)、一个把重复读实验结果横向拼接起来的csv文件(*readResult-combined.csv
)、一个把写结果和读结果拼接起来的csv文件(*allResult-combined.csv
)、一个把读结果取平均值并且按照不同粒度统计百分比的csv文件(*allResult-combined-processed.csv
其它类似,不再赘述。
二、实验结果
改变压缩方式
人工数据实验结果
234016.77980000002 us - 15.589238228946503% | 236135.9462 us - 15.56305048087338% | 235278.4135 us - 15.65358490817499% | 234469.6457 us - 15.537217392727715% | |
[Avg&Per] (D-2)8_3_timeDecoder_readLong(us) | 357893.9434 us - 23.841426880277478% | 360253.1425 us - 23.743262865502565% | 358341.1524 us - 23.84121675993312% | 363063.0738 us - 24.058508247673558% |
[Avg&Per] (D-2)8_4_valueDecoder_read(us) | 353821.6809 us - 23.570149451806063% | 359477.8899 us - 23.692168165422427% | 356440.4841 us - 23.714761161335133% | 355773.1939 us - 23.5754416723178% |
[Avg&Per] (D-2)8_5_checkValueSatisfyOrNot(us) | 223758.3096 us - 14.905861011513531% | 224938.2721 us - 14.825043539994219% | 224282.1638 us - 14.921980483485841% | 226053.6587 us - 14.979528915812129% |
[Avg&Per] (D-2)8_6_putIntoBatchData(us) | 328221.5562 us - 21.864774123677403% | 332305.597 us - 21.90131940694663% | 325251.7878 us - 21.639709317671908% | 325993.7043 us - 21.602092830601723% |
补充实验
时间戳和值都使用PLAIN编码,然后改变压缩。
RLCompressionRealExpScriptsRLCompressionSynExpScripts.sh
压缩方式 | GZIP | LZ4 | SNAPPY | UNCOMPRESSED | |||
dataset | synthetic | synthetic | synthetic | /disk/rl/zc_data/ZT11529.csv | /disk/rl/zc_data/ZT11529.csv | /disk/rl/zc_data/ZT11529.csv | /disk/rl/zc_data/ZT11529.csvsynthetic |
pagePointNum(ppn) | 10000 | 10000 | 10000 | 10000 | |||
numOfPagesInChunk(pic) | 10001001000 | 100 | 10001001000 | 100 | |||
chunksWritten(cw) | 101310 | 13 | 101310 | 13 | |||
timeEncoding(te) | TS_2DIFF | TS_2DIFF | TS_2DIFF | PLAIN | PLAIN | PLAIN | PLAINTS_2DIFF |
valueDataType(vt) | INT64DOUBLEINT64 | DOUBLE | INT64DOUBLEINT64 | DOUBLE | |||
valueEncoding(ve) | PLAIN | PLAIN | PLAIN | PLAIN | |||
compression(co) | GZIP | LZ4 | SNAPPY | UNCOMPRESSED | |||
totalPointNum | 10000000012780287100000000 | 12780287 | 10000000012780287100000000 | 12780287 | |||
tsfileSize(MB) | 76750.13128667578783 | 77074.844431971146107 | 76775.942390481546879 | 781195.42261510946026 | |||
chunkDataSize_stats_mean(MB) | 763.71300761977135579 | 775.08436436846096277 | 765.7941264932730039 | 7815.1421661426517868 | |||
compressedPageSize_stats_mean(B) | 8037541639.4186728917 | 8076461236.814447625 | 8046062145.4778918333 | 81874160003 | |||
uncompressedPageSize_stats_mean(B) | 8187416000381874 | 160003 | 8187416000381874 | 160003 | |||
timeBufferSize_stats_mean(B) | 1872800001872 | 80000 | 1872800001872 | 80000 | |||
valueBufferSize_stats_mean(B) | 80000 | 80000 | 80000 | 80000 | 80000 | ||
total_time(us) | 2096685.5981 | 1244193.0404 | 1195582.1173 | 1895095.38480000 | |||
[2] category: (A)get ChunkStatistic->(B)load on-disk Chunk->(C)get PageStatistics->(D)load in-memory PageData | |||||||
[Avg&Per] (A)get_chunkMetadatas | 10165486791.7566 3859 us - 04.9318259204986696%1844839300349514% | 86735100869.1136 9527 us - 18.1689451651044902%731470875680406%82918 | 99547.49919999999 05960000001 us - 17.0656107165107132%911992556094877%93896 | 88166.1409 6213 us - 14.2531889263513831%659966496653105% | |||
[Avg&Per] (B)load_on_disk_chunk | 5949593349328.8917000005 55549999996 us - 5416.53739687304131%84222128306944% | 4849630452828.197000001 32 us - 6539.35936296194448%197572537012675%5086618 | 450015.179 7795 us - 6535.36966894765757%76721916082827% | 48513861155773.484999999 8993 us - 6461.74924062031137%08737716190697% | |||
[Avg&Per] (C)get_pageHeader | 66132818.381399999991 293900000001 us - 0.060622054656159316%13587875585087916% | 71203913.158900000014 898699999998 us - 0.09595969816001626%3387935811871694% | 76923502.346500000014 114700000002 us - 0.09885667184764595%27834780402685505% | 76054450.696300000007 306 us - 0.10150975630087579%23521687180559223% | |||
[Avg&Per] (D_1)decompress_pageData | 28594281350175.0031000106 8349000001 us - 2665.211160404158996%09619618668371% | 521804261417.2723000004 90119999985 us - 722.032452670194604%628768326063632%605202 | 395144.8143000009 6617 us - 731.777644443672276%40606698493612% | 498170173785.42259999976 97769999996 us - 69.648853201570805%185299626198828% | |||
[Avg&Per] (D_2)decode_pageData | 1991910.319299996 us - 18.25899474764487% | 1954657.4198999994 us - 26.343279504596413% | 1998880.5966999987 us - 25.68821922031179% | 285009.94010000007 us - 13.74121984436101% | 336215.7518000001 us - 29.103394680056127% | 309969.7735 us - 24.636373494113883% | 469824.3798999997 us - 24.832139843435506% |
SUM | 2074124.0103 | 1155245.8243999998 | 1258179.389 | 1892001.18419999962041517.9070999995 us - 27.247207495465567% | |||
[3] D_1 compare each step inside | |||||||
[Avg&Per] (D-1)7_1_data_ByteBuffer_to_ByteArray(us) | 659524312.37819999998 080999999998 us - 20.5136549346918415%3365746911741179% | 10880910247.34350000018 3701 us - 595.6896105506002%205625954425964% | 1081329355.35939999981 254500000003 us - 432.622622294156905%648386471251286% | 11076533088.11740000003 566800000015 us - 6356.813731447511664%71920599957147% | |||
[Avg&Per] (D-1)7_2_data_decompress_PageDataByteArray(us) | 25546871274619.926599999 9271000002 us - 9799.36728361519128%48904214184739% | 68904183796.38600000006 93149999995 us - 3793.79892271446546%36815862249874% | 135345341381.91170000008 81159999996 us - 5496.601079805416944%64204980983628% | 5754721318.215800000035 842899999996 us - 3336.15396273496119%54397754446109% | |||
[Avg&Per] (D-1)7_3_data_ByteArray_to_ByteBuffer(us) | 811454.8335000000624 68920000000026 us - 0.03094155721322126%03549026028736633% | 1239659.949800000022 1047999999998 us - 0.680200047933345%334822790636471% | 1184604.7460000000272 4583000000001 us - 0.47794876167766753%17111658310904862%1229 | 973.1439000000355 5250000000001 us - 01.7081314098345313%6687808013713306% | |||
[Avg&Per] (D-1)7_4_data_split_time_value_Buffer(us) | 23121779.0582000000422 4489000000008 us - 0.08811989290364733%1388929066911369% | 33382148.2513999999974 4263999999994 us - 1.8312666870009688%091392632438828%3218 | 1902.3657999999955 0298000000007 us - 10.29834913874849%5384471358033915% | 40342956.201500000018 5653000000016 us - 25.3241744076926314%068035654596102% | |||
[3] D_2 compare each step inside | |||||||
[Avg&Per] (D-2)8_1_createBatchData(us) | 53843343.7852 3923 us - 0.053292019060348375%259976288059772% | 58483522.7599 578 us - 0.05759123169122766%27212097540020236% | 59133458.4963 9225 us - 0.058362326692940975%2672966393395521% | 60193730.3023 5984 us - 0.05943520403215091%2816327187407449% | |||
[Avg&Per] (D-2)8_2_timeDecoder_hasNext(us) | 1859842232202.2956 511 us - 18.406444711361424%05565768873081%1862234 | 231677.7849 2947 us - 1817.336946086748988%897191037883086%1864092 | 231911.3926 8133 us - 1817.397368271414525%921548782382853%1857778 | 237390.6739 8697 us - 1817.343895858133802%921263258420133% | |||
[Avg&Per] (D-2)8_3_timeDecoder_readLong(us) | 2074757254086.7936 67 us - 2019.533415498567617%75733129255225%2084700 | 255389.4377 3804 us - 2019.527508047369906%72896194244707%2063043 | 255976.8916 5501 us - 2019.360888969091857%78120978179259%2069930 | 261059.4964 1494 us - 2019.43870456313607%70804415658043% | |||
[Avg&Per] (D-2)8_4_valueDecoder_read(us) | 1876012241634.952 0221 us - 18.56648209392724%78903535624908% | 1881471242640.5433999998 7293 us - 18.526365490982297%744122040429612% | 1877809242328.2412 9765 us - 18.532744562964893%726560376256852% | 1876843246610.1276 5964 us - 18.53214021585961%6172847590372% | |||
[Avg&Per] (D-2)8_5_checkValueSatisfyOrNot(us) | 1780379230100.6374 908 us - 17.620020492363725%892240746329144% | 1780782231053.3133 5736 us - 17.534904586680103%849008259784295% | 1781949231043.2049 8535 us - 17.586668929952697%85447508020484% | 1780599235200.5789 0462 us - 17.5818216126948%755872210542634% | |||
[Avg&Per] (D-2)8_6_putIntoBatchData(us) | 2507922324669.0072 8983 us - 2425.82034518471963%24575862807894% | 2540605330206.1784 1447 us - 25.016684556527476%508595744055732% | 2539577329318.912 78729999997 us - 25.063966939883077%448909340023306% | 2536332340641.2055 1962 us - 25.044002546143567% |
715902896678855% |
- 可以看到,当时间戳列和值列都使用PLAIN编码之后,压缩负责了全部的压缩率,此时D-1操作和耗时占比有了明显提高;但是也可以看到,即便如此,除了GZIP之外的压缩方式的D-1耗时占比也没有增大到60%以上,D-2解码操作仍然有不小的基础耗时
改变值列编码方式
RLValueEncodingRealExpScriptsRLCompressionRealExpScripts.sh
ZT11529传感器数据如下图所示,共12,780,287个点。
编码方式 | GORILLA | PLAIN | RLE | TS_2DIFF | |||||||||
压缩方式 | GZIP | LZ4 | SNAPPY | UNCOMPRESSED | |||||||||
dataset | /disk/rl/zc_data/ZT11529.csv | /disk/rl/zc_data/ZT11529.csv | /disk/rl/zc_data/ZT11529.csv | /disk/rl/zc_data/ZT11529.csv | |||||||||
pagePointNum(ppn) | 10000 | 10000 | 10000 | 10000 | |||||||||
numOfPagesInChunk(pic) | 100 | 100 | 100 | 100 | |||||||||
chunksWritten(cw) | 13 | 13 | 13 | 13 | |||||||||
timeEncoding(te) | TS_2DIFF | TS_2DIFF | TS_2DIFF | TS_2DIFF | |||||||||
valueDataType(vt) | DOUBLE | DOUBLE | DOUBLE | DOUBLE | |||||||||
valueEncoding(ve) | GORILLAGORILLA | PLAIN | GORILLARLEGORILLA | TS_2DIFF | |||||||||
compression(co) | GZIPSNAPPY | LZ4SNAPPY | SNAPPY | UNCOMPRESSEDSNAPPY | |||||||||
totalPointNum | 12780287 | 12780287 | 12780287 | 12780287 | |||||||||
tsfileSize(MB) | 1923.348626141564121223 | 26.7774105159960175 | 2322.1564121210147858 | 3620.3077373598550797 | |||||||||
chunkDataSize_stats_mean(MB) | 1.5151396598131843411 | 2.86007007081166744 | 1.813184341740072568 | 21.837062438642661015 | |||||||||
compressedPageSize_stats_mean(B) | 1582418948.1666769 | 1944021758.27561667 | 1894818182.6929917 | 2968417160.75758925 | |||||||||
uncompressedPageSize_stats_mean(B) | 29684.757529684 | 91463.75754891729684 | 25245.75759341729684 | 22210.757536417 | |||||||||
timeBufferSize_stats_mean(B) | 11461.4625 | 11461.4625 | 11461.4625 | 11461.4625 | |||||||||
valueBufferSize_stats_mean(B) | 18221.2683318221.26833 | 80000 | 1822113782.2683344518221 | 10746.26833875 | |||||||||
[2] category: (A)get ChunkStatistic->(B)load on-disk Chunk->(C)get PageStatistics->(D)load in-memory PageData | |||||||||||||
[Avg&Per] (A)get_chunkMetadatas | 8731688753.85010000001 03020000001 us - 811.19883087518919%21274000219672% | 10028997256.6416 88870000001 us - 12.59103802007131%181751942467722% | 8946684509.6045 5108 us - 119.199530364576052%498496964807579% | 8876086370.777 7961 us - 1011.335699109318087%6687579647861% | |||||||||
[Avg&Per] (B)load_on_disk_chunk | 160699105215.6025 9241 us - 1513.089285299443361%29260306229143% | 176784138228.88239999997 6108 us - 2217.19476647997349%313597737139155% | 105802139816.9384 5536 us - 1315.244531050378352%714765088895252% | 19151896826.85280000002 5282 us - 2213.301305860612082%081335047881257% | |||||||||
[Avg&Per] (C)get_pageHeader | 24362352.5129999999995 1583 us - 0.22878239411203669%2971632563133493% | 21982625.3668000000007 5312999999983 us - 0.27599779517870476%328856613051254% | 23193388.9517000000005 6254999999987 us - 0.29041416798711567%38086658793698225% | 31343159.228800000001 255499999999 us - 0.3649635223062446%4268177375109232% | |||||||||
[Avg&Per] (D_1)decompress_pageData | 356587114457.5454999999 26139999992 us - 3314.482666568996265%460120522641777% | 31158182862.160799999983 35910000015 us - 322.911805656191469%904124612107367% | 115629155021.7179 76800000004 us - 1417.474658381255693%42376424722015% | 29640120684.983400000016 50350000002 us - 316.4515277590088287%304565026949902% | |||||||||
[Avg&Per] (D_2)decode_pageData | 457950480759.96670000016 0216999998 us - 4360.000434862259155%73737315655672% | 486085377408.0215000002 39589999994 us - 6147.02639204858503%27166909523449% | 485623506978.25309999986 10549999995 us - 6056.790866035802786%982107111140046% | 545723433147.8052999998 us - 63.54650374875476%3443000001 us - 58.51852422287182% | |||||||||
SUM | 791537.3956999998 us | 798381.7858000002 us | 889714.5634 us | 740188.4276000002 | |||||||||
[3] D_1 compare each step inside | |||||||||||||
[Avg&Per] (D-1)7_1_data_ByteBuffer_to_ByteArray(us) | 12691754.7615999999996 3673999999999 us - 01.36377892330196115%6860113426598728% | 18082320.5776999999991 3632000000002 us - 71.207505165727991%489795195953362% | 16871825.9916000000003 1960999999988 us - 1.412377327892232%548613686329812% | 31971607.6313 3271999999997 us - 411.86667970732365%758246074195737% | |||||||||
[Avg&Per] (D-1)7_2_data_decompress_PageDataByteArray(us) | 345856100527.23130000004 56879999992 us - 9996.08569249502276%61067644486589% | 21247151532.39440000001 73299999998 us - 8497.67466169480036%29198327791244%116100 | 113686.69619999993 73430000001 us - 9796.14384305311928%4591326329927% | 243287212.4552000000012 82 us - 3195.84820675254647%4016073295714% | |||||||||
[Avg&Per] (D-1)7_3_data_ByteArray_to_ByteBuffer(us) | 374431.18470000000025 9958999999999 us - 0.10720162531460037%41516388607230165% | 442468.10720000000003 2119000000003 us - 10.761876157051776%30061666178303303% | 424474.26030000000026 28859999999975 us - 0.35498732863644405%4024169332984032% | 421663.1930000000002 3730000000002 us - 50.51469221169019%7256599483773118% | |||||||||
[Avg&Per] (D-1)7_4_data_split_time_value_Buffer(us) | 15471340.4220999999993 373799999999 us - 01.443326956360674%288148326401935% | 15941429.8989000000004 1740000000002 us - 60.355956982419887%9176048643511714% | 13011873.2614999999998 7816000000007 us - 1.0887922903520593%5898367473790769% | 15861932.372500000001 989900000001 us - 202.770421328439692%1144866478555286% | |||||||||
[3] D_2 compare each step inside | [Avg&Per] (D-2)8_1_createBatchData(us) | 3430.8672 us - 0.22855030377902266% | 4174.8958 us - 0.275155541260775% | 3438.1465 us - 0.22874736939899784% | 3730.6205 us - 0.24721094086707432%inside | ||||||||
[Avg&Per] (D-2)8_21_timeDecoder_hasNextcreateBatchData(us) | 2340163348.77980000002 9259 us - 150.589238228946503%2218271110091239%236135 | 3442.9462 3069 us - 150.56305048087338%24318767981262365%235278 | 3466.4135 6887 us - 150.65358490817499%22277034162258696%234469 | 3338.6457 6453 us - 150.537217392727715%22194982158403212% | |||||||||
[Avg&Per] (D-2)8_32_timeDecoder_readLonghasNext(us) | 357893235317.9434 741 us - 2315.841426880277478%587043790733997%360253 | 244839.1425 8323 us - 2317.743262865502565%29713023052909%358341 | 239308.1524 8755 us - 2315.84121675993312%37805224577913% | 363063240223.0738 005 us - 2415.058508247673558%969786637750962% | |||||||||
[Avg&Per] (D-2)8_43_valueDecodertimeDecoder_readreadLong(us) | 353821361562.6809 5392 us - 23.570149451806063%949282819264262%359477 | 363364.8899 4154 us - 2325.692168165422427%67050285597614% | 356440362143.4841 5815 us - 23.714761161335133%271443255687238% | 355773357833.1939 1113 us - 23.5754416723178%788389623148674% | |||||||||
[Avg&Per] (D-2)8_54_valueDecoder_checkValueSatisfyOrNotread(us) | 223758356325.3096 4946 us - 1423.905861011513531%602389962111484%224938 | 241625.2721 5526 us - 1417.825043539994219%07005192367858%224282 | 396438.1638 5566 us - 1425.921980483485841%475247513902037%226053 | 335421.6587 7093 us - 1422.979528915812129%298501890735768% | |||||||||
[Avg&Per] (D-2)8_65_putIntoBatchDatacheckValueSatisfyOrNot(us) | 328221225554.5562 2633 us - 2114.864774123677403%940327764084076%332305 | 232364.597 86419999998 us - 2116.90131940694663%415814695306036%325251 | 224510.7878 63450000001 us - 2114.639709317671908%4271133273284%325993 | 230332.7043 4743 us - 2115.602092830601723% |
...
312273986066641% |
[Avg&Per] (D-2)8_6_putIntoBatchData(us) |
...
- 人工数据实验里发现另一个子步骤7_1_data_ByteBuffer_to_ByteArray(us)的占比也高,其主要因为人工数据实验里数据压缩率很低,从而子步骤7_2_data_decompress_PageDataByteArray(us)耗时少,从而相对来说7_1_data_ByteBuffer_to_ByteArray(us)占比高了。
327591.9399 us - 21.699128552797042% | 329856.9807 us - 23.30331261469754% | 330303.223 us - 21.225373315680617% | 337085.3345 us - 22.409098040713936% |
人工数据实验结果
改变压缩方式
RLCompressionSynExpScripts.sh
压缩方式 | GZIP | LZ4 | SNAPPY | UNCOMPRESSED |
...
改变编码方式
人工数据实验结果
RLValueEncodingSynExpScripts.sh
编码方式 | GORILLA | PLAIN | RLE | TS_2DIFF | ||
dataset | synthetic | synthetic | synthetic | synthetic | ||
pagePointNum(ppn) | 10000 | 10000 | 10000 | 10000 | ||
numOfPagesInChunk(pic) | 1000 | 1000 | 1000 | 1000 | ||
chunksWritten(cw) | 10 | 10 | 10 | 10 | ||
timeEncoding(te) | TS_2DIFF | TS_2DIFF | TS_2DIFF | TS_2DIFF | ||
valueDataType(vt) | INT64 | INT64 | INT64 | INT64 | ||
valueEncoding(ve) | GORILLAPLAIN | PLAIN | RLEPLAIN | TS_2DIFFPLAIN | ||
compression(co) | UNCOMPRESSEDGZIPUNCOMPRESSED | LZ4 | UNCOMPRESSEDSNAPPY | UNCOMPRESSED | ||
totalPointNum | 100000000 | 100000000 | 100000000 | 100000000 | ||
tsfileSize(MB) | 805767.38128951312866 | 781770.42261518444319781 | 767.84223189423904 | 793781.32447054226151 | ||
chunkDataSize_stats_mean(MB) | 76.71300761 | 8077.5380362408436436 | 7876.142166147941264 | 78.1841278114216614 | 79.33235168 | |
compressedPageSize_stats_mean(B) | 8438680375.2518941867 | 81874 | 81918 | 80764.81444 | 80460.47789 | 8187483122 |
uncompressedPageSize_stats_mean(B) | 84386.2518981874 | 81874 | 8191881874 | 8312281874 | ||
timeBufferSize_stats_mean(B) | 1872 | 1872 | 1872 | 1872 | ||
valueBufferSize_stats_mean(B) | 82512.2518980000 | 80000 | 8004480000 | 8124880000 | ||
[2] category: (A)get ChunkStatistic->(B)load on-disk Chunk->(C)get PageStatistics->(D)load in-memory PageData | ||||||
[Avg&Per] (A)get_chunkMetadatas | 91331101654.98490000001 7566 us - 0.8580518676474486%9318259204986696% | 10094486735.7581 1136 us - 1.2939556377951902%1689451651044902% | 8809882918.20449999999 49919999999 us - 01.9671805828234409%0656107165107132%88231 | 93896.2157 1409 us - 01.9257222461823116%2531889263513831% | ||
[Avg&Per] (B)load_on_disk_chunk | 55526455949593.935400001 8917000005 us - 5254.16637107440095%53739687304131% | 51701584849630.3812 197000001 us - 6665.27343223726832%35936296194448% | 52709145086618.364100001 179 us - 5765.866400973957255%36966894765757%5526099 | 4851386.6186 484999999 us - 5764.97985793317826%74924062031137% | ||
[Avg&Per] (C)get_pageHeader | 81856613.805399999992 381399999991 us - 0.07690455451459878%060622054656159316% | 77127120.402700000001 158900000014 us - 0.09886107156476356%09595969816001626% | 78137692.90999999998 346500000014 us - 0.08578451820695045%09885667184764595% | 77257605.986500000003 696300000007 us - 0.08106107934716146%10150975630087579% | ||
[Avg&Per] (D_1)decompress_pageData | 5481602859428.3352000009 0031000106 us - 526.149893543905802%211160404158996% | 525441521804.0348000005 2723000004 us - 67.7353412114264035%032452670194604%585036 | 605202.2351000007 8143000009 us - 67.422783415941812%777644443672276% | 632154498170.6457000006 42259999976 us - 6.632568914631739%648853201570805% | ||
[Avg&Per] (D_2)decode_pageData | 44437851991910.968300003 319299996 us - 4118.748778959531215%25899474764487% | 19969961954657.816400002 4198999994 us - 2526.598409841945312%343279504596413% | 31569021998880.088299994 5966999987 us - 3425.65785050907054%68821922031179%3276856 | 2041517.417400001 9070999995 us - 3427.38078982666053%247207495465567% | ||
[3] D_1 compare each step inside | ||||||
[Avg&Per] (D-1)7_1_data_ByteBuffer_to_ByteArray(us) | 11042165952.72189999989 37819999998 us - 642.93156378407389%5136549346918415% | 109687108809.91739999971 34350000018 us - 6359.92643734398307%6896105506002% | 113658108132.35759999987 35939999981 us - 6243.901942587101885%622622294156905% | 109187110765.47799999996 11740000003 us - 63.08420663361515%813731447511664% | ||
[Avg&Per] (D-1)7_2_data_decompress_PageDataByteArray(us) | 546242554687.26410000002 926599999 us - 3297.12084386602246%36728361519128% | 5709568904.555900000094 38600000006 us - 3337.27545607007074%79892271446546% | 62164135345.880499999985 91170000008 us - 3454.403908579311135%601079805416944%59072 | 57547.25890000005 215800000035 us - 3433.12961499817788%15396273496119% | ||
[Avg&Per] (D-1)7_3_data_ByteArray_to_ByteBuffer(us) | 1179811.3427000000347 8335000000624 us - 0.6934918640164246%03094155721322126% | 12341239.10400000004 949800000022 us - 0.7192394012210651%680200047933345% | 10321184.115900000022 7460000000272 us - 0.5712038820191121%47794876167766753% | 11931229.2719000000652 1439000000355 us - 0.6894253122110825%7081314098345313% | ||
[Avg&Per] (D-1)7_4_data_split_time_value_Buffer(us) | 38332312.2921999999994 0582000000422 us - 20.254100485887217%08811989290364733%3567 | 3338.0158000000133 2513999999974 us - 21.0788671847251163%8312666870009688%3835 | 3218.9775000000177 3657999999955 us - 21.122944951567873%29834913874849% | 36294034.1045000000054 201500000018 us - 2.0967530559958814%3241744076926314% | ||
[3] D_2 compare each step inside | ||||||
[Avg&Per] (D-2)8_1_createBatchData(us) | 60085384.9294 7852 us - 0.04720551599260821%053292019060348375% | 60055848.094 7599 us - 0.058953749593284185%05759123169122766% | 91365913.1988 4963 us - 0.07959246314858166%058362326692940975% | 62196019.250599999999 3023 us - 0.053213386787106826%05943520403215091% | ||
[Avg&Per] (D-2)8_2_timeDecoder_hasNext(us) | 17950671859842.4479 2956 us - 1418.10186066084482%406444711361424% | 18626311862234.8749 7849 us - 18.285997377780266%336946086748988% | 18051001864092.2815999999 3926 us - 1518.725618584694368%397368271414525%1838661 | 1857778.8702 6739 us - 1518.732028111177547%343895858133802% | ||
[Avg&Per] (D-2)8_3_timeDecoder_readLong(us) | 20736152074757.2138 7936 us - 1620.290102549307967%533415498567617% | 20894932084700.8765 4377 us - 20.513167449482335%527508047369906% | 20634692063043.3846 8916 us - 1720.976470800088325%360888969091857% | 21720292069930.5514 4964 us - 1820.58440125112089%43870456313607% | ||
[Avg&Per] (D-2)8_4_valueDecoder_read(us) | 46361951876012.4124 952 us - 3618.42146247963989%56648209392724% | 18803481881471.902 5433999998 us - 18.459930571697104%526365490982297% | 33524941877809.9242 2412 us - 2918.206164899804453%532744562964893% | 33645581876843.2458 1276 us - 2818.787960289219587%53214021585961% | ||
[Avg&Per] (D-2)8_5_checkValueSatisfyOrNot(us) | 17242391780379.435 6374 us - 1317.545443257159627%620020492363725% | 17842051780782.2864 3133 us - 17.516060810611705%534904586680103% | 17231281781949.802 2049 us - 1517.01150190311584%586668929952697%1780807 | 1780599.3196 5789 us - 1517.237010821076382%5818216126948% | ||
[Avg&Per] (D-2)8_6_putIntoBatchData(us) | 2507922.0072 us - 24.82034518471963% | 2540605.1784 2494168.5891 us - 1925.59392553705509%016684556527476% | 25634252539577.3348 912 us - 25.165890040835308%063966939883077% | 25253932536332.9444 2055 us - 22.000651349148434% | 2525103.5280999998 us - 21.605386140618513% |
- 总的来说,使用的人工数据集不太好,其随机生成的取值,编码压缩效率不高,甚至PLAIN编码的空间是最小的
RLValueEncodingRealExpScripts.sh
ZT11529传感器数据如下图所示,共12,780,287个点。
25.044002546143567% |
改变值列编码方式
RLValueEncodingSynExpScripts.sh
编码方式编码方式GORILLAPLAINRLETS_2DIFF | GORILLA | PLAIN | RLE | TS_2DIFF | |||
dataset | /disk/rl/zc_data/ZT11529.csv | /disk/rl/zc_data/ZT11529.csv | /disk/rl/zc_data/ZT11529.csv | synthetic | synthetic | synthetic | synthetic/disk/rl/zc_data/ZT11529.csv |
pagePointNum(ppn) | 10000 | 10000 | 10000 | 10000 | |||
numOfPagesInChunk(pic) | 1001000100 | 1000 | 1001000100 | 1000 | |||
chunksWritten(cw) | 131013 | 10 | 131013 | 10 | |||
timeEncoding(te) | TS_2DIFF | TS_2DIFF | TS_2DIFF | TS_2DIFF | |||
valueDataType(vt) | DOUBLEINT64DOUBLE | INT64 | DOUBLEINT64DOUBLE | INT64 | |||
valueEncoding(ve) | GORILLA | PLAIN | RLE | TS_2DIFF | |||
compression(co) | SNAPPYUNCOMPRESSEDSNAPPY | UNCOMPRESSED | SNAPPYUNCOMPRESSEDSNAPPY | UNCOMPRESSED | |||
totalPointNum | 1278028710000000012780287 | 100000000 | 1278028710000000012780287 | 100000000 | |||
tsfileSize(MB) | 23805.156412123812895 | 26781.599601754226151 | 22781.101478588422318 | 20793.985507973244705 | |||
chunkDataSize_stats_mean(MB) | 180.81318434153803624 | 278.08116674414216614 | 178.740072568184127811 | 79.64266101533235168 | |||
compressedPageSize_stats_mean(B) | 18948.69 | 21758.61667 | 18182.29917 | 84386.25189 | 81874 | 81918 | 8312217160.8925 |
uncompressedPageSize_stats_mean(B) | 2968484386.757525189 | 91463.48917 | 25245.93417 | 81874 | 81918 | 8312222210.36417 | |
timeBufferSize_stats_mean(B) | 11461.4625 | 11461.4625 | 11461.4625 | 1872 | 1872 | 1872 | 187211461.4625 |
valueBufferSize_stats_mean(B) | 1822182512.2683325189 | 80000 | 13782.445 | 80044 | 8124810746.875 | ||
[2] category: (A)get ChunkStatistic->(B)load on-disk Chunk->(C)get PageStatistics->(D)load in-memory PageData | |||||||
[Avg&Per] (A)get_chunkMetadatas | 8875391331.03020000001 98490000001 us - 110.21274000219672%8580518676474486% | 97256100944.88870000001 7581 us - 121.181751942467722%2939556377951902% | 8450988098.5108 20449999999 us - 90.498496964807579%9671805828234409% | 8637088231.7961 2157 us - 110.6687579647861%9257222461823116% | |||
[Avg&Per] (B)load_on_disk_chunk | 1052155552645.9241 935400001 us - 1352.29260306229143%16637107440095% | 1382285170158.6108 3812 us - 1766.313597737139155%27343223726832% | 1398165270914.5536 364100001 us - 1557.714765088895252%866400973957255% | 968265526099.5282 6186 us - 1357.081335047881257%97985793317826% | |||
[Avg&Per] (C)get_pageHeader | 23528185.1583 805399999992 us - 0.2971632563133493%07690455451459878% | 26257712.5312999999983 402700000001 us - 0.328856613051254%09886107156476356% | 33887813.6254999999987 90999999998 us - 0.38086658793698225%08578451820695045% | 31597725.255499999999 986500000003 us - 0.4268177375109232%08106107934716146% | |||
[Avg&Per] (D_1)decompress_pageData | 114457548160.26139999992 3352000009 us - 145.460120522641777%149893543905802% | 182862525441.35910000015 0348000005 us - 226.904124612107367%7353412114264035% | 155021585036.76800000004 2351000007 us - 176.42376424722015%422783415941812% | 120684632154.50350000002 6457000006 us - 166.304565026949902%632568914631739% | |||
[Avg&Per] (D_2)decode_pageData | 4807594443785.0216999998 968300003 us - 6041.73737315655672%748778959531215% | 3774081996996.39589999994 816400002 us - 4725.27166909523449%598409841945312% | 5069783156902.10549999995 088299994 us - 5634.982107111140046%65785050907054% | 4331473276856.3443000001 417400001 us - 5834.51852422287182%38078982666053% | |||
[3] D_1 compare each step inside | |||||||
[Avg&Per] (D-1)7_1_data_ByteBuffer_to_ByteArray(us) | 1754110421.3673999999999 72189999989 us - 164.6860113426598728%93156378407389%2320 | 109687.3632000000002 91739999971 us - 163.489795195953362%92643734398307%1825 | 113658.1960999999988 35759999987 us - 162.548613686329812%901942587101885%1607 | 109187.3271999999997 47799999996 us - 163.758246074195737%08420663361515% | |||
[Avg&Per] (D-1)7_2_data_decompress_PageDataByteArray(us) | 10052754624.56879999992 26410000002 us - 9632.61067644486589%12084386602246% | 15153257095.73299999998 555900000094 us - 9733.29198327791244%27545607007074% | 11368662164.73430000001 880499999985 us - 9634.4591326329927%403908579311135% | 8721259072.82 25890000005 us - 9534.4016073295714%12961499817788% | |||
[Avg&Per] (D-1)7_3_data_ByteArray_to_ByteBuffer(us) | 4311179.9958999999999 3427000000347 us - 0.41516388607230165%6934918640164246% | 4681234.2119000000003 10400000004 us - 0.30061666178303303%7192394012210651% | 4741032.28859999999975 115900000022 us - 0.4024169332984032%5712038820191121% | 6631193.3730000000002 2719000000652 us - 0.7256599483773118%6894253122110825% | |||
[Avg&Per] (D-1)7_4_data_split_time_value_Buffer(us) | 13403833.373799999999 2921999999994 us - 12.288148326401935%254100485887217% | 14293567.1740000000002 0158000000133 us - 02.9176048643511714%0788671847251163% | 18733835.7816000000007 9775000000177 us - 12.5898367473790769%122944951567873% | 19323629.989900000001 1045000000054 us - 2.1144866478555286%0967530559958814% | |||
[3] D_2 compare each step inside | |||||||
[Avg&Per] (D-2)8_1_createBatchData(us) | 33486008.9259 9294 us - 0.2218271110091239%04720551599260821% | 34426005.3069 094 us - 0.24318767981262365%058953749593284185% | 34669136.6887 1988 us - 0.22277034162258696%07959246314858166% | 33386219.6453 250599999999 us - 0.22194982158403212%053213386787106826% | |||
[Avg&Per] (D-2)8_2_timeDecoder_hasNext(us) | 2353171795067.741 4479 us - 1514.587043790733997%10186066084482% | 2448391862631.8323 8749 us - 1718.29713023052909%285997377780266% | 2393081805100.8755 2815999999 us - 15.37805224577913%725618584694368% | 2402231838661.005 8702 us - 15.969786637750962%732028111177547% | |||
[Avg&Per] (D-2)8_3_timeDecoder_readLong(us) | 3615622073615.5392 2138 us - 2316.949282819264262%290102549307967% | 3633642089493.4154 8765 us - 2520.67050285597614%513167449482335% | 3621432063469.5815 3846 us - 2317.271443255687238%976470800088325%357833 | 2172029.1113 5514 us - 2318.788389623148674%58440125112089% | |||
[Avg&Per] (D-2)8_4_valueDecoder_read(us) | 3563254636195.4946 4124 us - 2336.602389962111484%42146247963989% | 2416251880348.5526 902 us - 1718.07005192367858%459930571697104% | 3964383352494.5566 9242 us - 2529.475247513902037%206164899804453% | 3354213364558.7093 2458 us - 2228.298501890735768%787960289219587% | |||
[Avg&Per] (D-2)8_5_checkValueSatisfyOrNot(us) | 2255541724239.2633 435 us - 1413.940327764084076%545443257159627% | 2323641784205.86419999998 2864 us - 1617.415814695306036%516060810611705% | 2245101723128.63450000001 802 us - 1415.4271133273284%01150190311584% | 2303321780807.4743 3196 us - 15.312273986066641%237010821076382% | |||
[Avg&Per] (D-2)8_6_putIntoBatchData(us) | 3275912494168.9399 5891 us - 2119.699128552797042%59392553705509% | 3298562563425.9807 3348 us - 2325.30331261469754%165890040835308% | 3303032525393.223 9444 us - 2122.225373315680617%000651349148434%337085 | 2525103.3345 5280999998 us - 2221.409098040713936%605386140618513% |