THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
Lookup can be optimized using Bloom filters in the future.
Computing individual state changes
...
- Bind journal lifecycle to entry / version
- Avoid extra hash lookup
- Easily access state and track changes
List state
For a list, if only appends were made, it’s enough to store its length right after the previous snapshot to get the added elements. Otherwise, if it was truncated for example, the full list is snapshotted.
...
Hardware: 6x m5.xlarge: 1 JM + 1 source + 4 operators (holding map state). Each TM had 12 GB RAM.
File full | File incremental | Rocks Incremental | |
Job runtime | 12m22s | 9m21s | 18m |
Checkpoint size, Mb (75/95%) | 9565 | 114/9565 | 1681/6142 |
Checkpoint duration, s (75/95%) | 16 | 1/15 | 21/49 |
Sync duration, ms (75/95/99/100%) | 0/1.65 | 0/1/6/199 | 488/1960 |
Note that the use case is specifically targeted at incremental backends:
...