...
RLValueEncodingSynExpScripts.sh
压缩方式GZIPLZ4SNAPPYUNCOMPRESSEDTS_2DIFF |
dataset | synthetic | synthetic | synthetic | synthetic |
pagePointNum(ppn) | 10000 | 10000 | 10000 | 10000 |
numOfPagesInChunk(pic) | 1000 | 1000 | 1000 | 1000 |
chunksWritten(cw) | 10 | 10 | 10 | 10 |
timeEncoding(te) | TS_2DIFF | TS_2DIFF | TS_2DIFF | TS_2DIFF |
valueDataType(vt) | INT64 | INT64 | INT64 | INT64 |
valueEncoding(ve) |
compressionGORILLA | PLAIN | RLE | TS_2DIFF |
compression(co) | UNCOMPRESSED | UNCOMPRESSED | UNCOMPRESSED | UNCOMPRESSED |
totalPointNum | 100000000 | 100000000 | 100000000 | 100000000 |
tsfileSize(MB) | 805.3812895 | 781.4226151 | 781.8422318 | 793.3244705 |
chunkDataSize_stats_mean(MB) | 80.53803624 | 78.14216614 | 78.18412781 | 79.33235168 |
compressedPageSize_stats_mean(B) | 84386.25189 | 81874 | 81918 | 83122 |
uncompressedPageSize_stats_mean(B) |
timeBufferSize84386.25189 | 81874 | 81918 | 83122 |
timeBufferSize_stats_mean(B) | 1872 | 1872 | 1872 | 1872 |
valueBufferSize_stats_mean(B) | 82512.25189 | 80000 | 80044 | 81248 |
[2] category: (A)get ChunkStatistic->(B)load on-disk Chunk->(C)get PageStatistics->(D)load in-memory PageData |
[Avg&Per] (A)get_chunkMetadatas | 91331.98490000001 us - 0.8580518676474486% | 100944.7581 us - 1.2939556377951902% | 88098.20449999999 us - 0.9671805828234409% | 88231.2157 us - 0.9257222461823116% |
[Avg&Per] (B)load_on_disk_chunk |
[Avg&5552645.935400001 us - 52.16637107440095% | 5170158.3812 us - 66.27343223726832% | 5270914.364100001 us - 57.866400973957255% | 5526099.6186 us - 57.97985793317826% |
[Avg&Per] (C)get_pageHeader | 8185.805399999992 us - 0.07690455451459878% | 7712.402700000001 us - 0.09886107156476356% | 7813.90999999998 us - 0.08578451820695045% | 7725.986500000003 us - 0.08106107934716146% |
[Avg&Per] (D_1) |
decompress_pageDatadecompress_pageData | 548160.3352000009 us - 5.149893543905802% | 525441.0348000005 us - 6.7353412114264035% | 585036.2351000007 us - 6.422783415941812% | 632154.6457000006 us - 6.632568914631739% |
[Avg&Per] (D_2)decode_pageData | 4443785.968300003 us - 41.748778959531215% | 1996996.816400002 us - 25.598409841945312% | 3156902.088299994 us - 34.65785050907054% | 3276856.417400001 us - 34.38078982666053% |
[3] D_1 compare each step inside |
[Avg&Per] (D-1)7_1_data_ByteBuffer_to_ByteArray(us) |
|
|
|
|
[Avg&Per] (D-1)7_2_data_decompress_PageDataByteArray(us) |
|
|
|
|
[Avg&Per] (D-1)7_3_data_ByteArray_to_ByteBuffer(us) |
|
|
|
|
[Avg&Per] (D-1)7_4_data_split_time_value_Buffer(us) |
|
|
|
|
[3] D_2 compare each step inside |
[Avg&Per] (D-2)8_1_createBatchData(us) |
|
|
|
|
[Avg&Per] (D-2)8_2_timeDecoder_hasNext(us) |
|
|
|
|
[Avg&Per] (D-2)8_3_timeDecoder_readLong(us) |
|
|
|
|
[Avg&Per] (D-2)8_4_valueDecoder_read(us) |
|
|
|
|
[Avg&Per] (D-2)8_5_checkValueSatisfyOrNot(us) |
|
|
|
|
[Avg&Per] (D-2)8_6_putIntoBatchData(us) |
|
|
|
|
- 总的来说,使用的人工数据集不太好,其随机生成的取值,编码压缩效率都不高
中车数据实验结果
RLValueEncodingRealExpScripts.sh
ZT11529传感器数据如下图所示,共12,780,287个点。
Image Added
编码方式GORILLAPLAINRLETS_2DIFF | GORILLA | PLAIN | RLE | TS_2DIFF |
压缩方式 | GZIP | LZ4 | SNAPPY | UNCOMPRESSED |
dataset |
|
|
|
|
pagePointNum(ppn) |
|
|
|
|
numOfPagesInChunk(pic) |
|
|
|
|
chunksWritten(cw) |
|
|
|
|
timeEncoding(te) |
|
|
|
|
valueDataType(vt) |
|
|
|
|
valueEncoding(ve) |
|
|
|
|
compression(co) |
|
|
|
|
totalPointNum |
|
|
|
|
tsfileSize(MB) |
|
|
|
|
chunkDataSize_stats_mean(MB) |
|
|
|
|
compressedPageSize_stats_mean(B) |
|
|
|
|
uncompressedPageSize_stats_mean(B) |
|
|
|
|
timeBufferSize_stats_mean(B) |
|
|
|
|
valueBufferSize_stats_mean(B) |
|
|
|
|
[2] category: (A)get ChunkStatistic->(B)load on-disk Chunk->(C)get PageStatistics->(D)load in-memory PageData |
[Avg&Per] (A)get_chunkMetadatas |
|
|
|
|
[Avg&Per] (B)load_on_disk_chunk |
|
|
|
|
[Avg&Per] (C)get_pageHeader |
|
|
|
|
[Avg&Per] (D_1)decompress_pageData |
|
|
|
|
[Avg&Per] (D_2)decode_pageData |
|
|
|
|
[3] D_1 compare each step inside |
[Avg&Per] (D-1)7_1_data_ByteBuffer_to_ByteArray(us) |
|
|
|
|
[Avg&Per] (D-1)7_2_data_decompress_PageDataByteArray(us) |
|
|
|
|
[Avg&Per] (D-1)7_3_data_ByteArray_to_ByteBuffer(us) |
|
|
|
|
[Avg&Per] (D-1)7_4_data_split_time_value_Buffer(us) |
|
|
|
|
[3] D_2 compare each step inside |
[Avg&Per] (D-2)8_1_createBatchData(us) |
|
|
|
|
[Avg&Per] (D-2)8_2_timeDecoder_hasNext(us) |
|
|
|
|
[Avg&Per] (D-2)8_3_timeDecoder_readLong(us) |
|
|
|
|
[Avg&Per] (D-2)8_4_valueDecoder_read(us) |
|
|
|
|
[Avg&Per] (D-2)8_5_checkValueSatisfyOrNot(us) |
|
|
|
|
[Avg&Per] (D-2)8_6_putIntoBatchData(us) |
|
|
|
|