...
RLValueEncodingSynExpScripts.sh
压缩方式GZIPLZ4SNAPPYUNCOMPRESSED| TS_2DIFF |
| dataset | synthetic | synthetic | synthetic | synthetic |
| pagePointNum(ppn) | 10000 | 10000 | 10000 | 10000 |
| numOfPagesInChunk(pic) | 1000 | 1000 | 1000 | 1000 |
| chunksWritten(cw) | 10 | 10 | 10 | 10 |
| timeEncoding(te) | TS_2DIFF | TS_2DIFF | TS_2DIFF | TS_2DIFF |
| valueDataType(vt) | INT64 | INT64 | INT64 | INT64 |
| valueEncoding(ve) |
compression| GORILLA | PLAIN | RLE | TS_2DIFF |
| compression(co) | UNCOMPRESSED | UNCOMPRESSED | UNCOMPRESSED | UNCOMPRESSED |
| totalPointNum | 100000000 | 100000000 | 100000000 | 100000000 |
| tsfileSize(MB) | 805.3812895 | 781.4226151 | 781.8422318 | 793.3244705 |
| chunkDataSize_stats_mean(MB) | 80.53803624 | 78.14216614 | 78.18412781 | 79.33235168 |
| compressedPageSize_stats_mean(B) | 84386.25189 | 81874 | 81918 | 83122 |
| uncompressedPageSize_stats_mean(B) |
timeBufferSize| 84386.25189 | 81874 | 81918 | 83122 |
| timeBufferSize_stats_mean(B) | 1872 | 1872 | 1872 | 1872 |
| valueBufferSize_stats_mean(B) | 82512.25189 | 80000 | 80044 | 81248 |
| [2] category: (A)get ChunkStatistic->(B)load on-disk Chunk->(C)get PageStatistics->(D)load in-memory PageData |
| [Avg&Per] (A)get_chunkMetadatas | 91331.98490000001 us - 0.8580518676474486% | 100944.7581 us - 1.2939556377951902% | 88098.20449999999 us - 0.9671805828234409% | 88231.2157 us - 0.9257222461823116% |
| [Avg&Per] (B)load_on_disk_chunk |
[Avg&| 5552645.935400001 us - 52.16637107440095% | 5170158.3812 us - 66.27343223726832% | 5270914.364100001 us - 57.866400973957255% | 5526099.6186 us - 57.97985793317826% |
| [Avg&Per] (C)get_pageHeader | 8185.805399999992 us - 0.07690455451459878% | 7712.402700000001 us - 0.09886107156476356% | 7813.90999999998 us - 0.08578451820695045% | 7725.986500000003 us - 0.08106107934716146% |
| [Avg&Per] (D_1) |
decompress_pageData| decompress_pageData | 548160.3352000009 us - 5.149893543905802% | 525441.0348000005 us - 6.7353412114264035% | 585036.2351000007 us - 6.422783415941812% | 632154.6457000006 us - 6.632568914631739% |
| [Avg&Per] (D_2)decode_pageData | 4443785.968300003 us - 41.748778959531215% | 1996996.816400002 us - 25.598409841945312% | 3156902.088299994 us - 34.65785050907054% | 3276856.417400001 us - 34.38078982666053% |
| [3] D_1 compare each step inside |
| [Avg&Per] (D-1)7_1_data_ByteBuffer_to_ByteArray(us) |
|
|
|
|
| [Avg&Per] (D-1)7_2_data_decompress_PageDataByteArray(us) |
|
|
|
|
| [Avg&Per] (D-1)7_3_data_ByteArray_to_ByteBuffer(us) |
|
|
|
|
| [Avg&Per] (D-1)7_4_data_split_time_value_Buffer(us) |
|
|
|
|
| [3] D_2 compare each step inside |
| [Avg&Per] (D-2)8_1_createBatchData(us) |
|
|
|
|
| [Avg&Per] (D-2)8_2_timeDecoder_hasNext(us) |
|
|
|
|
| [Avg&Per] (D-2)8_3_timeDecoder_readLong(us) |
|
|
|
|
| [Avg&Per] (D-2)8_4_valueDecoder_read(us) |
|
|
|
|
| [Avg&Per] (D-2)8_5_checkValueSatisfyOrNot(us) |
|
|
|
|
| [Avg&Per] (D-2)8_6_putIntoBatchData(us) |
|
|
|
|
- 总的来说，使用的人工数据集不太好，其随机生成的取值，编码压缩效率都不高
中车数据实验结果
RLValueEncodingRealExpScripts.sh
ZT11529传感器数据如下图所示，共12,780,287个点。
Image Added
| 编码方式GORILLAPLAINRLETS_2DIFF | GORILLA | PLAIN | RLE | TS_2DIFF |
压缩方式 | GZIP | LZ4 | SNAPPY | UNCOMPRESSED |
| dataset |
|
|
|
|
| pagePointNum(ppn) |
|
|
|
|
| numOfPagesInChunk(pic) |
|
|
|
|
| chunksWritten(cw) |
|
|
|
|
| timeEncoding(te) |
|
|
|
|
| valueDataType(vt) |
|
|
|
|
| valueEncoding(ve) |
|
|
|
|
| compression(co) |
|
|
|
|
| totalPointNum |
|
|
|
|
| tsfileSize(MB) |
|
|
|
|
| chunkDataSize_stats_mean(MB) |
|
|
|
|
| compressedPageSize_stats_mean(B) |
|
|
|
|
| uncompressedPageSize_stats_mean(B) |
|
|
|
|
| timeBufferSize_stats_mean(B) |
|
|
|
|
| valueBufferSize_stats_mean(B) |
|
|
|
|
| [2] category: (A)get ChunkStatistic->(B)load on-disk Chunk->(C)get PageStatistics->(D)load in-memory PageData |
| [Avg&Per] (A)get_chunkMetadatas |
|
|
|
|
| [Avg&Per] (B)load_on_disk_chunk |
|
|
|
|
| [Avg&Per] (C)get_pageHeader |
|
|
|
|
| [Avg&Per] (D_1)decompress_pageData |
|
|
|
|
| [Avg&Per] (D_2)decode_pageData |
|
|
|
|
| [3] D_1 compare each step inside |
| [Avg&Per] (D-1)7_1_data_ByteBuffer_to_ByteArray(us) |
|
|
|
|
| [Avg&Per] (D-1)7_2_data_decompress_PageDataByteArray(us) |
|
|
|
|
| [Avg&Per] (D-1)7_3_data_ByteArray_to_ByteBuffer(us) |
|
|
|
|
| [Avg&Per] (D-1)7_4_data_split_time_value_Buffer(us) |
|
|
|
|
| [3] D_2 compare each step inside |
| [Avg&Per] (D-2)8_1_createBatchData(us) |
|
|
|
|
| [Avg&Per] (D-2)8_2_timeDecoder_hasNext(us) |
|
|
|
|
| [Avg&Per] (D-2)8_3_timeDecoder_readLong(us) |
|
|
|
|
| [Avg&Per] (D-2)8_4_valueDecoder_read(us) |
|
|
|
|
| [Avg&Per] (D-2)8_5_checkValueSatisfyOrNot(us) |
|
|
|
|
| [Avg&Per] (D-2)8_6_putIntoBatchData(us) |
|
|
|
|