Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.


Discussion thread-
Vote thread-
JIRA

Jira
serverASF JIRA
serverId5aa69414-a9e9-3523-82ec-879b028fb15b
keyFLINK-32285

Releaseml-2.3.0


Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

...

Flink ML relies on the Flink public APIs to provide machine learning algorithms, running as a library on an API-compatible Flink cluster. However, different companies may have different versions of Flink clusters in production, so we would better to support Flink ML being built with different versions of Flink to run on various versions of Flink clusters (at least the latest few stable versions).

...

  • In the following releases, we will publish three artifacts for each module to support running on the latest three Flink versions.
  • The artifactId of the Jar built by Flink ML will have the name of the corresponding Flink version. Below lists the modified artifactId when built with Flink 1.17.

Original Artifact ID

New Artifact ID

flink-ml-benchmark

flink-ml-benchmark-1.17

flink-ml-core

flink-ml-core-1.17

flink-ml-dist

flink-ml-dist-1.17

flink-ml-examples

flink-ml-examples-1.17

flink-ml-iteration

flink-ml-iteration-1.17

flink-ml-lib

flink-ml-lib-1.17

flink-ml-python

flink-ml-python-1.17

flink-ml-servable-core

flink-ml-servable-core-1.17

flink-ml-servable-lib

flink-ml-servable-lib-1.17

flink-ml-tests

flink-ml-tests-1.17

flink-ml-uber

flink-ml-uber-1.17

Proposed Changes

The basic idea is to make the Flink ML relies on a default Flink version, but can also be built and run with two previous Flink versions by setting profiles. Here is the POC of the FLIP:  https://github.com/jiangxin369/flink-ml/pull/2.

...

  • For those modules that have incompatible classes with the default Flink version, copy the classes to a separate module and replace the incompatible APIs with the correct ones. The copied class should have the same fully qualified name to overwrite the original class. Currently,  only flinkonly flink-ml-iteration module has incompatible API usages when built with Flink 1.15 and 1.17 (default version), so just split the flink-ml-iteration module to flinkiteration module has incompatible API usages when built with Flink 1.15 and 1.17 (default version), so just split the flink-ml-iteration module to flink-ml-iteration-common(stores the default classes) and flinkstores the default classes) and flink-ml-iteration-1.15(stores the copied and modified classes), the latter relies on the former.

...

Replaces all artifactId of the Jars to be published to maven central with Flink version bound.

  • stores the copied and modified classes), the latter relies on the former. Apache Paimon also supports multiple Flink versions in this way.
  • Replaces all artifactId of the Jars to be published to maven central with Flink version bound.
  • Adds
  • maven profiles to set the Flink version during the maven building, e.g. "mvn package -Pflink1.15"  would compile and build the artifact with Flink 1.15. If no profile is passed, it would be built with the default version, i.e., Flink 1.7 for now.

Compatibility, Deprecation, and Migration Plan

...