Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Status

...

Page properties


Discussion thread

...

...

Jira
serverASF JIRA
serverId5aa69414-a9e9-3523-82ec-879b028fb15b
keyFLINK-25718

...

Release1.15


Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

...

   @Experimental
    public static final ConfigOption<String> PYTHON_EXECUTION_MODE =
        ConfigOptions.key("python.execution-mode")
                .stringType()
                .defaultValue("process")
                .withDescription(
                        "Specify the python runtime execution mode. The optional values are `process`, `multi-thread` and `sub-interpreter`. "
                                + "The `process` mode means that the Python user-defined functions will be executed in separate Python process. "
                                + "The `multi-thread` mode means that the Python user-defined functions will be executed in the same thread as Java Operator, but it will be affected by GIL performance. "
                                + "The `sub-interpreter` mode means that the Python user-defined functions will be executed in python different sub-interpreters rather than different threads of one interpreter, "
                                + "which can largely overcome the effects of the GIL, but it maybe fail in some CPython extensions libraries, such as numpy, tensorflow. "
                                + "Note that if the python operator dose not support `multi-thread` and `sub-interpreter` mode, we will still use `process` mode.");

We will introduce a new Python Configuration `python.execution-mode`,  which is used to specify the python runtime execution mode. The possible values are `process` and `thread`. The `process` mode means that the Python user-defined functions will be executed in a separate Python Process and it is the current PyFlink Runtime execution mode. The `thread` mode means that the Python user-defined functions will be executed in the same thread as Java Operator, which is the new execution mode we will discuss in this FLIP.

Proposed Changes

The architecture of Process Mode

...