...
- support for re-writing warm content to an SSD if read from spinning media
- this is uses the SSD as an additional level in the hierarchy between RAM and HDD
- support for only writing the second miss in an SSD-only environment
The Interim Caching Layer
When we thinking about the storage of ATS, the original desigin is to support many disks, with the same size(best work with block devices without raid), and build up the volumes(partitions), then assign each of the volumes to some domain(hostname), we find out that if we don't make big change in the storage design, we don't have a easy way to archive the multi-tiered caching storage, then we come to the following INTERIM solution:
- we think that if your deployed mixed storage, the fast device is less. IE, 1/10 of the slow devices, in size or in counts.
- we assume that in most case, 8 fast disks is fairly many.
- we assume your slow disk should not go beyond 32TB per disk.
- the slow devices holding all the data, should not loss them when fast disk fails.
- compare to the storage percent, ie 10%, the fast devices may loss data during the server restart.
- the fast disks should balance to all the slow disks, in size and even in IOPS. the volumes is building upon the slow disks, with slices from every disks, we should spread the load on each fast disk too.
we make a design with balance:
- a interim cache of the cache, which is lived on the fast disks(SSD in most case), will loss the data if you restart the server process. make the data on interim caching device a real interim.
- we support 8 fast disks at most
- we make a block level interim cache, which does not contains any index info
what we have done:
- we steal 4bit from the Dir struct, to identify each of the fast disks
- we cut down the up limit of the disk in size to 32TB
- we split each fast disks into volumes confiured in the volume.config, and bind them.
- we store the interim data in the dir on the slow storages
- write data from the origin server on the slow disks
- we setup a interal LRU list, limited in size(1M bucket), and bump the busy blocks into fast device
- we do hot data evocation on the interim caching devices, to make it more efficent, we do compare the blocks with the LRU list
- the interim cache device(fast disks), works with RRD writing too.
coins and pins:
- coins:
- we make a solid solution without big change in the storage architecture
- the LUR help us get the hot blocks, it is efficent
- the block layer interim caching can help on small objects and big ones too
- pins:
- loss data if the server process crash
- limited in block device only for used as interim caching device
- the interim caching device space is not a add-on, but a copy of the hot data on the slow devices
- we set lower the max disk size of the storage from 0.5PB to 32TB.
- the interim caching function is not enabled by default configuration
codes & structs:
the change of Dir:
Code Block |
---|
@@ -155,15 +157,42 @@ struct FreeDir
unsigned int reserved:8;
unsigned int prev:16; // (2)
unsigned int next:16; // (3)
+#if TS_USE_INTERIM_CACHE == 1
+ unsigned int offset_high:12; // 8GB * 4K = 32TB
+ unsigned int index:3; // interim index
+ unsigned int ininterim:1; // in interim or not
+#else
inku16 offset_high; // 0: empty
+#endif
#else
uint16_t w[5];
FreeDir() { dir_clear(this); }
|
we split the stat of read_success into disk, interim and ram:
Code Block |
---|
@@ -2633,6 +2888,11 @@ register_cache_stats(RecRawStatBlock *rsb, const char *prefix)
REG_INT("read.active", cache_read_active_stat);
REG_INT("read.success", cache_read_success_stat);
REG_INT("read.failure", cache_read_failure_stat);
+ REG_INT("interim.read.success", cache_interim_read_success_stat);
+ REG_INT("disk.read.success", cache_disk_read_success_stat);
+ REG_INT("ram.read.success", cache_ram_read_success_stat);
REG_INT("write.active", cache_write_active_stat);
REG_INT("write.success", cache_write_success_stat);
REG_INT("write.failure", cache_write_failure_stat);
|
how to enable interim caching:
configure to enable the interim caching function:
add '--enable-interim-cache' option.
we introduced two record config:
- proxy.config.cache.interim.storage:
disk device(s) for the interim cache, we only support block devices in full path, multipule disks may seperate by SPACE. example: LOCAL proxy.config.cache.interim.storage STRING /dev/sdb /dev/sdc1 - proxy.config.cache.interim.migrate_threshold:
controls how many times one content should be migrate to the interim cache from the storage, which will control the writing on the interim disk. default to '2', which means it should be seen in the LRU list two times before we thinking of migrate it to the interim caching device.
some of the result:
we have a system with 160G SSD + 3 * 500G SAS, 16G RAM, 4 cores, here is the result of tsar and iostat -x:
```
Code Block |
---|
Time --------------------ts------------------ -------------ts_cache-----------
Time qps cons Bps rt rpc hit ramhit band ssdhit
24/06/13-10:30 901.83 18.89 22.6M 17.36 47.74 87.30 68.08 88.90 22.49
24/06/13-10:35 934.12 18.88 22.0M 14.34 49.47 87.60 68.53 90.70 22.21
24/06/13-10:40 938.14 18.92 21.7M 15.36 49.58 87.70 68.02 89.50 22.45
avg-cpu: %user %nice %system %iowait %steal %idle
5.47 0.00 15.62 25.09 0.00 53.82
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 7.33 25.67 3.33 1600.00 1438.00 104.76 0.45 15.46 12.17 35.30
sdb 0.00 0.00 28.67 11.33 1461.00 8723.00 254.60 0.74 18.47 11.21 44.83
sdc 0.00 0.00 25.67 2.00 2178.00 1373.33 128.36 0.40 14.05 11.04 30.53
sdd 0.00 0.00 196.00 4.00 14790.00 2823.00 88.06 0.13 0.66 0.41 8.30
|
```