[Deprecated] AIP-93 Technical Implementation

Technical Details

Stage 1 - Building the generic StateVariable model

Building the async-aware StateVariable model would require minimal lift. Like the Variable model, this would be “unrelated” to other models, making the creation a bit easier. Such model should be usable by various Airflow components, other than triggers. Therefore, it should provide an abstraction layer that 

This model contains two core fields: key and value. Below is the pseudo-code for the StateVariable model. 

class StateVariable:
    """
    StateVariable class allows various methods fetching and storing values.
    """
	key: str
	value: str

    def set(self, key, value):
        # Using db connection as an example here, this method will vary
        db_backend.set_value(key=key, value=value)

    def get(self):
        return db_backend.get_value(key=key)

Despite their similarities, StateVariable will differ from Variable in a number of ways. These include:

• StateVariable will NOT use a traditional Secrets Backend. Instead, it will have its own type of backend.
• StateVariable will have its own UI component(s), and can be integrated more "natively" into a DAG/Asset Watcher.
• StateVariable will have a sort of "owner" field, something that Variable  does not have.

Stage 2 - Extending the backend of StateVariable

The state model will be reusable and extendable in other Airflow components, such as task, worker, scheduler, etc. It should support not only the API, but also the Task SDK for fetching and storing the value for different components.

Stage 3 - Implement the state-enabled Trigger/Asset Watcher

Once the StateVariable  model has been created, it can be used within a BaseEventTrigger  just like Variable  would; using StateVariable.get()  and StateVariable.set(...) . This pattern is quite intuitive for DAG authors.

It would require some sort of get and set logic to be implemented in a class that inherits from BaseEventTrigger in order to store and retrieve watermark value. This could be used in a standalone manner, or be implemented in the form of a "helper" method in the BaseEventTrigger class.