Page History

...

• Add IntermediateDataSetID to StreamTransformation and Operator
• TableEnvironment stores the Table → (IntermediateResultId, [ClusterPartitionDescriptor]) mapping
• Client replaces the source of the cached node before optimization.
• Client adds a sink to the node that should be cached.

...

Impact of optimization

To explain how the optimizer affects the cache node, let look at a very simple DAG, where one scan node followed by a filter node. The optimizer can push the filter to the scan node such that the scan node will produce less data to the downstream node. However, such optimization will affect the result of the scan node, which is an undesired behavior if we want to cache the scan node.

...