...

Discussion thread: To be added

JIRA: To be added 

Released: 1.15

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

...

The existing Flink ML library allows users to compose an Estimator/Transformer from a pipeline (i.e. linear sequence) of Estimator/Transformer.

The following use-cases are not supported yet. And we would like to address these use-cases with the changes proposed in this FLIP.

1) Compose a directed-acyclic-graph (i.e DAG) Estimator/Transformer into an Estimator/Transformer.

. Users only need to construct this Pipeline once and generate the corresponding PipelineModel, without having to explicitly construct the fitted PipelineModel as a linear sequence of stages.

However, in the use-case that needs a DAG of Estimator/Transformer, users currently needs to separately build the DAG separately, once for the training logic and once for the inference logic. This experience is inferior to the cases supported by the Pipeline.

To improve the user experience, we propose to add several helper classes that allow users to compose Estimator/Transformer/AlgoOperator from For example, the workflow may involve the join of 2 tables, where each table could be generated by a chain of Estimator/Transformer. The entire workflow is therefore a DAG of Estimator/Transformer/AlgoOperator.

Public Interfaces

This FLIP proposes to add the Graph, GraphTransformer and GraphBuilder classes. The following code block shows the public APIs of these classes.

...