DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
| Info |
|---|
AIP-93 has shifted directions a bit; rather than creating a state-store for “Asset watching”, we’re planning on creating a general, async-aware state-store that can be accessed from “anywhere” in Airflow (Scheduler, Triggerer, Workers, etc.) |
ProcessState Model
The content of this page is now deprecated. With the work being done in AIP-103, this is no longer necessary. |
StateVariable Model
The StateVariableThe ProcessState model should be relatively straightforward. This This model will be accessible via some sort of API/Task SDK. It will almost act as a sort of key-value store. One of the things that’s important to me is making sure that there is uniqueness outside of just specifying a “key”. i.e. some sort of “hash” or "process_name“owner". This model will very closely mimic the Variable modelwould something like this.
| Code Block | ||||||
|---|---|---|---|---|---|---|
| ||||||
class ProcessState(Base)StateVariable: id = ... # Automatically created created_date = ... update_date = ... process_nameowner = ... key = ... value = ... # Implement set, update, delete, get methods to store and retrieve state def set(): ... def update(): ... def delete(): ... def get(): ... |
API/Task SDK
Again, mimicking the VariableThe StateVariable model , the ProcessState model will be accessible via the API and the Task SDK. This will allow for process state to be update each Airflow "component" (Workers, Triggerer, Scheduler/DAG Processor).
Other Considerations
Naming has been a challenge thus far. Some options have included State, StateVariable, ProcessState, Watermark, and more. For now, we're going with StateVariable.