Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  • 测试时间序列ID化后内存空间占用
  • 测试SHA256 java版本性能


实验场景

每时间序列标识长度为200字节,共有100万时间序列每时间序列标识长度为200字节,共有100万时间序列,其中measurement占用20字节

每个时间序列通过SHA256算法ID化成32字节的ID,使用哈希表存储,键为ID,值为数据类型及磁盘指针

...

Code Block
languagejava
import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
import java.util.Random;

public class SpeedMain {
  private static MessageDigest md;
  private static final int size = 1_000_000;
  private static final int pathSize = 200;

  public static void main(String[] args) throws NoSuchAlgorithmException {
    md = MessageDigest.getInstance("SHA-256");
    StringBuilder path = new StringBuilder();
    Random random = new Random();

    for (int i = 0; i < pathSize; i++) {
      path.append(random.nextInt(9));
    }

    long curTime = System.nanoTime();

    for(int i = 0; i < size; i++){
      cal(path.toString());
      md.reset();
    }

    long finishTime = System.nanoTime();
    System.out.println("Throughput: " + size / ((finishTime - curTime) / 1_000_000) + " num / ms.");

  }

  private static byte[] cal(String s){
    return md.digest(s.getBytes());
  }
}


新实现下的内存占用

测试数据量

100万设备,每个设备下100个测点10万设备,每个设备下100个测点

数据结构

HashMap<DeviceID, HashMap<String, SchemaEntry>>

...

内存占用
Hashmap Node3.2GB
Hashmap Node[]1.2GB

DeviceID

48MB
HashMap48MB
Schema Entry4GB

String

2.4G
char[]53.6G2G*
总计1614.496GB096GB

*java中char类型占用两个字节,长度为20的string占用40字节,外加8字节引用,4字节数组长度和4字节地址对齐java中char类型占用两个字节,长度为10的string占用20字节,外加8字节引用,4字节数组长度