Discussion thread-
Vote thread-
JIRA

Unable to render Jira issues macro, execution error.

Releaseml-2.3.0


Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Flink ML relies on the Flink public APIs to provide machine learning algorithms, running as a library on an API-compatible Flink cluster. However, different companies may have different versions of Flink clusters in production, so we would better support Flink ML being built with different versions of Flink to run on various versions of Flink clusters (at least the latest few stable versions).

Public Interfaces

  • In the following releases, we will publish three artifacts for each module to support running on the latest three Flink versions.
  • The artifactId of the Jar built by Flink ML will have the name of the corresponding Flink version. Below lists the modified artifactId when built with Flink 1.17.

Original Artifact ID

New Artifact ID

flink-ml-benchmark

flink-ml-benchmark-1.17

flink-ml-core

flink-ml-core-1.17

flink-ml-dist

flink-ml-dist-1.17

flink-ml-examples

flink-ml-examples-1.17

flink-ml-iteration

flink-ml-iteration-1.17

flink-ml-lib

flink-ml-lib-1.17

flink-ml-python

flink-ml-python-1.17

flink-ml-servable-core

flink-ml-servable-core-1.17

flink-ml-servable-lib

flink-ml-servable-lib-1.17

flink-ml-tests

flink-ml-tests-1.17

flink-ml-uber

flink-ml-uber-1.17

Proposed Changes

The basic idea is to make the Flink ML relies on a default Flink version, but can also be built and run with two previous Flink versions by setting profiles. Here is the POC of the FLIP:  https://github.com/jiangxin369/flink-ml/pull/2.

To summarize code changes:

  • For those modules that have incompatible classes with the default Flink version, copy the classes to a separate module and replace the incompatible APIs with the correct ones. The copied class should have the same fully qualified name to overwrite the original class. Currently, only flink-ml-iteration module has incompatible API usages when built with Flink 1.15 and 1.17 (default version), so just split the flink-ml-iteration module to flink-ml-iteration-common(stores the default classes) and flink-ml-iteration-1.15(stores the copied and modified classes), the latter relies on the former. Apache Paimon also supports multiple Flink versions in this way.
  • Replaces all artifactId of the Jars to be published to maven central with Flink version bound.
  • Adds maven profiles to set the Flink version during the maven building, e.g. "mvn package -Pflink1.15"  would compile and build the artifact with Flink 1.15. If no profile is passed, it would be built with the default version, i.e., Flink 1.7 for now.

Compatibility, Deprecation, and Migration Plan

Users need to replace the artifactId during migration.

  • No labels