THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
- TimerService needs to access user states so triggered timer can clean up expired user states.
- User states need to access TimerService to register timers when certain event TtlUpdateEvent happened to user states
...
- .
...
- We can add callback functions to all state implementations to ensure that we get notified on all state accesses.
Thus the initial step should be moving TimerService to state backends to facilitate the above proposal.
Currently, there are still two issues:
- Where should the code live in?
- Should it live with operators, state backends, or both?
- We may also need something in operators or state backends (e.g. operators/StreamingRuntimeContext/KeyedStateStore) to have access to TimerService
...
- How to distinguish TTL timers from user timers so that they don't interfere with each other.
- we need a good strategy to ensure that the TTL timers do not interfere with user-configured timers, i.e., what happens if a user configures a timer for the same time as a Ttl timer, and which one should be invoked first? The main problem right now is that we have to distinguish between user and TTL timers. Currently, timer service does not support timer tags (or another method) to distinguish timers.
Test Plan
Normal unit, integration, and end-to-end tests
...
- Rocksdb’s built-in TTL only support TTL on record creation, not able to extend/refresh TTL upon read and/or write
- It only supports processing time, and don’t doesn’t support event time.
- The TTL logic heavily rely on state backend implementation. If we add more state backends and rely on their internal feature for TTL, it’s hard to unify their behaviors and will introduce unnecessary system complexity
- Using external feature to build Flink’s TTL makes migration and compatibility hard