Page History

...

(4) RLCompressionExpScripts.sh：在不同的压缩方式参数下（UNCOMPRESSED, SNAPPY, GZIP, LZ4），写TsFile，清空系统缓存，然后进行读TsFile实验。LZ4），写TsFile，清空系统缓存，然后进行若干次重复读TsFile实验，把读实验结果进行汇总，把写文件的空间结果和读文件的耗时结果汇总到一起，最后对读实验结果进行平均值和百分比统计计算。

输入：
- 工具地址：
  - WRITE_READ_JAR_PATH：RLTsFileReadCostBench-0.13.1-jar-with-dependencies.jar的地址
  - Calculator_JAR_PATH：把若干次重复读实验结果进行平均值和百分比统计计算的RLRepeatReadResultAvgPercCalculator-0.13.1-jar-with-dependencies.jar的地址
  - TOOL_PATH：用于替换脚本中变量值的自动脚本工具RLtool.sh的地址
  - READ_SCRIPT_PATH：RLReadExpScripts.sh的地址
- 写数据参数：见RLTsFileReadCostBench写数据参数
- 读数据参数：见RLTsFileReadCostBench读数据参数
- REPEAT：读实验重复次数
输出：不同压缩方式下的一个TsFile文件、一个TsFile空间统计结果文件（ *writeResult.csv）、REPEAT个读TsFile耗时结果csv文件（ *readResult-T*csv）、一个把重复读实验结果横向拼接起来的csv文件（*readResult-combined.csv）、一个把写结果和读结果拼接起来的csv文件（*allResult-combined.csv）、一个把读结果取平均值并且按照不同粒度统计百分比的csv文件（ *allResult-combined-processed.csv）

...

ZT11529传感器数据如下图所示，共12,780,287个点。

压缩方式	GZIP	LZ4	SNAPPY	UNCOMPRESSED
dataset	/disk/rl/zc_data/ZT11529.csv

syntheticsynthetic

/disk/rl/zc_data/ZT11529.csv

synthetic


pagePointNum(ppn)	10000	10000	10000	10000
numOfPagesInChunk(pic)	100

1000

100

1000

100

1000

100
chunksWritten(cw)	13

10

13

10

13

10

13
timeEncoding(te)	TS_2DIFF	TS_2DIFF	TS_2DIFF	TS_2DIFF
valueDataType(vt)	DOUBLE

INT64

DOUBLE

INT64

DOUBLE

INT64

DOUBLE
valueEncoding(ve)	GORILLA

PLAIN

GORILLA

PLAIN

GORILLA

PLAIN

GORILLA
compression(co)	GZIP	LZ4	SNAPPY	UNCOMPRESSED
totalPointNum	12780287

100000000

12780287

100000000

12780287

100000000

12780287
tsfileSize(MB)	19.34862614

770

23.

8444319

77741051

767

23.

9423904

15641212

781

36.

4226151

30773735
chunkDataSize_stats_mean(MB)	1.515139659

77

1.

08436436

86007007

76

1.

7941264

813184341

78

2.

14216614

837062438
compressedPageSize_stats_mean(B)	15824.16667

80764

19440.

81444

275

80460

18948.

47789

69

81874

29684.7575
uncompressedPageSize_stats_mean(B)	29684.7575

8187481874

29684.7575

81874


timeBufferSize_stats_mean(B)	11461.4625

18721872

11461.4625

1872


valueBufferSize_stats_mean(B)	18221.26833

8000080000

18221.26833

80000


[1] each step
[Avg&Per] (A)1_index_read_deserialize_MagicString_FileMetadataSize(us)

26642

10294.

8733

8866 us - 0.

24422388846191903%

9666637540862395%

11918

17480.

6528

267 us -

0

2.

16062988779113208%

1945906166045948%

10188

10013.

2737

4112 us -

0

1.

1309325873262339%

2534900973847278%

10953

12118.

8906

2366 us -

0

1.

14619657707769018%

4111012934589997%

[Avg&Per] (A)2_index_read_deserialize_IndexRootNode_MetaOffset_BloomFilter(us)

5777

5024.

7715

5339 us - 0.

05296237408352104%

4717909959598364%

5484

8883.

9663

9004 us -

0

1.

07392190510886776%

115344774578661%

5140

4736.

9507

353 us - 0.

0660679126109081%

5929020055841159%

6219

5782.

5857

8723 us - 0.

08300997092132265%

6733833355290506%

[Avg&Per] (A)3_2_index_read_deserialize_IndexRootNode_exclude_to_TimeseriesMetadata_forExactGet(us)

69234

71997.

1118

4296 us -

0

6.

6346396579532295%

760376125143112%

69331

73925.

4945

4742 us -

0

9.

9343933722044904%

281102628888053%

67589

74716.

2748

8403 us -

0

9.

8686102165735712%

353138261607208%

76722

70859.

6646

6681 us -

1

8.

0239823783523703%

251214480330036%

[Avg&Per] (B)4_data_read_deserialize_ChunkHeader(us)

8684

8457.

7625

0026 us - 0.

07960952425196037%

7940911055429293%

10008

4862.

712599999999

6532 us - 0.

13488927052826724%

6104902793831642%

4487

4758.

9059

252799999999 us - 0.

05767543633654741%

5956434472253724%

7069

2203.

0819

0548 us - 0.

09434780888370882%

25653348589027036%

[Avg&Per] (B)5_data_read_ChunkData(us)

5940909

152242.

1292

5999 us -

54

14.

457787348789346%

295194193900432%

4839621

171922.

4844

22919999997 us -

65

21.

22447369141621%

584276200590327%

5082130

101044.

2731

6856 us -

65

12.

31199351132103%

648887603152982%

4844317

189315.

4031

798 us -

64

22.

65489281142766%

04477237472181%

[Avg&Per] (C)6_data_deserialize_PageHeader(us)

6613

2436.

381399999991

5129999999995 us - 0.

060622054656159316%

22878239411203669%

7120

2198.

158900000014

3668000000007 us - 0.

09595969816001626%

27599779517870476%

7692

2319.

346500000014

9517000000005 us - 0.

09885667184764595%

29041416798711567%

7605

3134.

696300000007

228800000001 us - 0.

10150975630087579%

3649635223062446%

[Avg&Per] (D-1)7_data_decompress_PageData(us)

2859428

356587.

0031000106

5454999999 us -

26

33.

211160404158996%

482666568996265%

521804

31158.

2723000004

160799999983 us -

7

3.

032452670194604%

911805656191469%

605202

115629.

8143000009

7179 us -

7

14.

777644443672276%

474658381255693%

498170

29640.

42259999976

983400000016 us -

6

3.

648853201570805%

4515277590088287%

[Avg&Per] (D-2)8_data_decode_PageData(us)

1991910

457950.

319299996

96670000016 us -

18

43.

25899474764487%

000434862259155%

1954657

486085.

4198999994

0215000002 us -

26

61.

343279504596413%

02639204858503%

1998880

485623.

5966999987

25309999986 us -

25

60.

68821922031179%

790866035802786%

2041517

545723.

9070999995

8052999998 us -

27

63.

247207495465567%

54650374875476%
[2] category: (A)get ChunkStatistic->(B)load on-disk Chunk->(C)get PageStatistics->(D)load in-memory PageData
[Avg&Per] (A)get_chunkMetadatas

101654

87316.

7566

85010000001 us -

0

8.

9318259204986696%

19883087518919%

86735

100289.

1136

6416 us -

1

12.

1689451651044902%

59103802007131%

82918

89466.

49919999999

6045 us -

1

11.

0656107165107132%

199530364576052%

93896

88760.

1409

777 us -

1

10.

2531889263513831%

335699109318087%

[Avg&Per] (B)load_on_disk_chunk

5949593

160699.

8917000005

6025 us -

54

15.

53739687304131%

089285299443361%

4849630

176784.

197000001

88239999997 us -

65

22.

35936296194448%

19476647997349%

5086618

105802.

179

9384 us -

65

13.

36966894765757%

244531050378352%

4851386

191518.

484999999

85280000002 us -

64

22.

74924062031137%

301305860612082%

[Avg&Per] (C)get_pageHeader

6613

2436.

381399999991

5129999999995 us - 0.

060622054656159316%

22878239411203669%

7120

2198.

158900000014

3668000000007 us - 0.

09595969816001626%

27599779517870476%

7692

2319.

346500000014

9517000000005 us - 0.

09885667184764595%

29041416798711567%

7605

3134.

696300000007

228800000001 us - 0.

10150975630087579%

3649635223062446%

[Avg&Per] (D_1)decompress_pageData

2859428

356587.

0031000106

5454999999 us -

26

33.

211160404158996%

482666568996265%

521804

31158.

2723000004

160799999983 us -

7

3.

032452670194604%

911805656191469%

605202

115629.

8143000009

7179 us -

7

14.

777644443672276%

474658381255693%

498170

29640.

42259999976

983400000016 us -

6

3.

648853201570805%

4515277590088287%

[Avg&Per] (D_2)decode_pageData

1991910

457950.

319299996

96670000016 us -

18

43.

25899474764487%

000434862259155%

1954657

486085.

4198999994

0215000002 us -

26

61.

343279504596413%

02639204858503%

1998880

485623.

5966999987

25309999986 us -

25

60.

68821922031179%

790866035802786%

2041517

545723.

9070999995

8052999998 us -

27

63.

247207495465567%

54650374875476%
[3] D_1 compare each step inside
[Avg&Per] (D-1)7_1_data_ByteBuffer_to_ByteArray(us)

65952.37819999998 us - 2.5136549346918415%108809.34350000018 us - 59.6896105506002%108132.35939999981 us - 43.622622294156905%110765.11740000003 us - 63.813731447511664%


[Avg&Per] (D-1)7_2_data_decompress_PageDataByteArray(us)
[Avg&Per] (D-1)7_

2

3_data_ByteArray_

decompress

to_

PageDataByteArray

ByteBuffer(us)

2554687.926599999 us - 97.36728361519128%68904.38600000006 us - 37.79892271446546%135345.91170000008 us - 54.601079805416944%57547.215800000035 us - 33.15396273496119%


[Avg&Per] (D-1)7_4_data_split_time_value_Buffer(us)
[3] D_2 compare each step inside

[Avg&Per] (D-1)7_3_data_ByteArray_to_ByteBuffer(us)811.8335000000624 us - 0.03094155721322126%1239.949800000022 us - 0.680200047933345%1184.7460000000272 us - 0.47794876167766753%1229.1439000000355 us - 0.7081314098345313%[Avg&Per] (D-1)7_4_data_split_time_value_Buffer(us)2312.0582000000422 us - 0.08811989290364733%3338.2513999999974 us - 1.8312666870009688%3218.3657999999955 us - 1.29834913874849%4034.201500000018 us - 2.3241744076926314%[3] D_2 compare each step inside[Avg&Per] (D-2)8_1_createBatchData(us)5384.7852 us - 0.053292019060348375%5848.7599 us - 0.05759123169122766%5913.4963 us - 0.058362326692940975%6019.3023 us - 0.05943520403215091%[Avg&Per] (D-2)8_2_timeDecoder_hasNext(us)1859842.2956 us - 18.406444711361424%1862234.7849 us - 18.336946086748988%1864092.3926 us - 18.397368271414525%1857778.6739 us - 18.343895858133802%[Avg&Per] (D-2)8_3_timeDecoder_readLong(us)2074757.7936 us - 20.533415498567617%2084700.4377 us - 20.527508047369906%2063043.8916 us - 20.360888969091857%2069930.4964 us - 20.43870456313607%


[Avg&Per] (D-2)8_

4_valueDecoder_read(us)1876012.952 us - 18.56648209392724%1881471.5433999998 us - 18.526365490982297%1877809.2412 us - 18.532744562964893%1876843.1276 us - 18.53214021585961%

1_createBatchData(us)
[Avg&Per] (D-2)8_

5

2_timeDecoder_

checkValueSatisfyOrNot

hasNext(us)

1780379.6374 us - 17.620020492363725%1780782.3133 us - 17.534904586680103%1781949.2049 us - 17.586668929952697%


[Avg&Per] (D-2)8_3_timeDecoder_readLong(us)
[Avg&Per] (D-2)8_4_valueDecoder_read(us)
[Avg&Per] (D-2)8_5_checkValueSatisfyOrNot(us)

1780599.5789 us - 17.5818216126948%


[Avg&Per] (D-2)8_6_putIntoBatchData(us)

2507922.0072 us - 24.82034518471963%2540605.1784 us - 25.016684556527476%2539577.912 us - 25.063966939883077%2536332.2055 us - 25.044002546143567%

Space shortcuts

Page tree

Versions Compared

Old Version 9

New Version 10

Key

改变编码方式

人工数据实验结果

中车数据实验结果