...
Previously Flink supported bounded iteration with DataSet API and supported the unbounded iteration with DataStream API. However, since Flink aims to deprecate the DataSet API and the iteration in the DataStream API is rather incomplete, thus we would require to re-implement a new iteration library in the Flink-ml repository to support the algorithms.
Besides, the previous DataStream and DataSet iteration APIs also have some caveats to support algorithm implementation:
- Lack of deterministic termination detection and checkpoint support for the DataStream iteration.
- Lack of the support for multiple inputs, arbitrary outputs and nested iteration for both iteration APIs.
- Lack of asynchronous iteration support for the DataSet iteration.
- The current
Overall Design
To reduce the development and maintenance overhead, it would be preferred to have a unified implementation for different types of iterations. In fact, the different iteration types shares the same requirements in runtime implementation:
...