Page History

...

In this case, the consuming job will throw an exception and the job will fail. As a result, PartitionTracker in ResourceManager will release all the cluster partitions that are impacted(implemented in FLIP-67). The TableEnvironment will fell back and resubmit the original DAG without using the cache. The original DAG will run as an ordinary job that follows the existing recovery strategy. Note that because there is no cache available, the TableEnvironment (planner) will again create a Sink to cache the result that was initially cached, therefore the cache will be recreated after the execution of the original DAG.

The above process is transparent to the users.

Users can specify the maximum number of retries before giving up by the following configuration.

cache.retries.max

When the retries maximum is reached, an exception is thrown and it is up to the users to handle the failure.

Changes needed

In order to implement the default intermediate result storage, the following changes are needed.

...