Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

写入速度 / 文件所含内容原始文件仅保留起止时间统计信息去除全部统计信息去除全部统计信息及measurementID
写入时间44.858 s44.216 s42.582 s4059.714 s
写入时间比100 % 98.57 %94.93 %90.76 %

每序列1个点 (SNAPPY)

写入速度 / 文件所含内容原始文件仅保留起止时间统计信息去除全部统计信息去除全部统计信息及measurementID
写入时间42.771 s42.401 s41.987 s4055.030 s
写入时间比100 %99.13 %98.17 %93.59 %

每序列10个点 (GZIP)

写入速度 / 文件所含内容原始文件仅保留起止时间统计信息去除全部统计信息去除全部统计信息及measurementID
写入时间81.053 s79.042 s74.134 s70.287 s
写入时间比100 % 97.52 %91.46 %86.72 %

...

写入速度 / 文件所含内容原始文件仅保留起止时间统计信息去除全部统计信息去除全部统计信息及measurementID
写入时间79.911 s72.701 s65.384 s63.224 s
写入时间比100 %90.98 %81.82 %79.12 %


实验结论

(1)存储空间:每序列10个点的情况下,移除统计信息大约能节省33%的空间;每序列1个点的情况下,移除统计信息大约能节省50%的空间

(2)写入速度:去除统计信息可以节省10-20%的写入时间 

实验源码

写入程序

Code Block
import java.io.File;
import java.util.Random;
import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
import org.apache.iotdb.tsfile.file.metadata.enums.TSEncoding;
import org.apache.iotdb.tsfile.fileSystem.FSFactoryProducer;
import org.apache.iotdb.tsfile.read.common.Path;
import org.apache.iotdb.tsfile.write.TsFileWriter;
import org.apache.iotdb.tsfile.write.record.TSRecord;
import org.apache.iotdb.tsfile.write.record.datapoint.DataPoint;
import org.apache.iotdb.tsfile.write.record.datapoint.LongDataPoint;
import org.apache.iotdb.tsfile.write.schema.UnaryMeasurementSchema;

/**
 * An example of writing data with TSRecord to TsFile It uses the interface: public void
 * addMeasurement(MeasurementSchema measurementSchema) throws WriteProcessException
 */
public class TsFileWriteTest {
  public static int deviceNum;
  public static int sensorNum;
  public static int fileNum;
  public static int pointNum = 10;

  public static void main(String[] args) {
    try {
      deviceNum = 1; // Integer.parseInt(cl.getOptionValue("d"));
      sensorNum = 1; // Integer.parseInt(cl.getOptionValue("m"));
      fileNum = 1; // Integer.parseInt(cl.getOptionValue("f"));
    } catch (Exception e) {
      e.printStackTrace();
    }

    for (int fileIndex = 0; fileIndex < fileNum; fileIndex++) {
      try {
        String path =
            "/Users/SilverNarcissus/iotdb/tsfile_test"
                + "/withoutStat/"
                + deviceNum
                + "."
                + sensorNum
                + "/test"
                + fileIndex
                + ".tsfile";
        File f = FSFactoryProducer.getFSFactory().getFile(path);
        if (f.exists()) {
          f.delete();
        }

        try {
          TsFileWriter tsFileWriter = new TsFileWriter(f);
          for (int i = 1; i <= deviceNum; i++) {
            for (int j = 1; j <= sensorNum; j++) {
              Path path1 = new Path(Constant.DEVICE_PREFIX + i);
              tsFileWriter.registerTimeseries(
                  path1,
                  new UnaryMeasurementSchema(
                      Constant.SENSOR_ + j, TSDataType.INT64, TSEncoding.RLE));
            }
          }
          // construct TSRecord
          int count = 0;
          for (int j = 1; j <= deviceNum; j++) {
            for (int i = 1; i <= pointNum; i++) {
              TSRecord tsRecord = new TSRecord(i, Constant.DEVICE_PREFIX + j);
              for (int t = 1; t <= sensorNum; t++) {
                DataPoint dPoint1 =
                    new LongDataPoint(Constant.SENSOR_ + t, new Random().nextLong());
                tsRecord.addTuple(dPoint1);
                count++;
              }
              // write TSRecord
              tsFileWriter.write(tsRecord);
              if (count % 100000 == 0) {
                System.out.println(count);
                tsFileWriter.flushAllChunkGroups();
              }
            }
          }
          tsFileWriter.close();
        } catch (Throwable e) {
          e.printStackTrace();
          System.out.println(e.getMessage());
        }

      } catch (Throwable e) {
        e.printStackTrace();
        System.out.println(e.getMessage());
      }
    }
  }
}

...