Status
...
...
...
| Jira |
---|
server | ASF JIRA |
---|
serverId | 5aa69414-a9e9-3523-82ec-879b028fb15b |
---|
key | FLINK-25718 |
---|
|
|
---|
|
...
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
...
@Experimental public static final ConfigOption<String> PYTHON_EXECUTION_MODE = ConfigOptions.key("python.execution-mode") .stringType() .defaultValue("process") .withDescription( "Specify the python runtime execution mode. The optional values are `process`, `multi-thread` and `sub-interpreter`. " + "The `process` mode means that the Python user-defined functions will be executed in separate Python process. " + "The `multi-thread` mode means that the Python user-defined functions will be executed in the same thread as Java Operator, but it will be affected by GIL performance. " + "The `sub-interpreter` mode means that the Python user-defined functions will be executed in python different sub-interpreters rather than different threads of one interpreter, " + "which can largely overcome the effects of the GIL, but it maybe fail in some CPython extensions libraries, such as numpy, tensorflow. " + "Note that if the python operator dose not support `multi-thread` and `sub-interpreter` mode, we will still use `process` mode."); |
We will introduce a new Python Configuration `python.execution-mode`, which is used to specify the python runtime execution mode. The possible values are `process` and `thread`. The `process` mode means that the Python user-defined functions will be executed in a separate Python Process and it is the current PyFlink Runtime execution mode. The `thread` mode means that the Python user-defined functions will be executed in the same thread as Java Operator, which is the new execution mode we will discuss in this FLIP.
Proposed Changes
The architecture of Process Mode
...