This page includes tips and troubleshooting information regarding Apache Beam's multi-language pipelines framework. For full documentation on multi-language pipelines, please see here.

Table of Contents

"java: command not found" when starting the pipeline

...

"KeyError: 'beam:coders:javasdk:0.1'" when expanding the transform using expansion service

This occurs due to Java expansion service returning an expanded transform that uses a Java specific coder as one of its outputs. Cross-language transforms require coders used at the SDK boundary to be Beam Standard Coders that can be interpreted by all SDKs. Note that internal sub-transforms of the expanded transforms may choose Java specify coders. What matters are final outputs produced by the expanded Java transform.

The solution will be to update the user's transform to produce output PCollection types that use standard coders at the SDK boundaries.

"java.lang.IllegalArgumentException: Unknown Coder URN beam:coder:pickled_python:v1" when runner a Python pipeline

This usually means that an expansion request that is sent from Python SDK to a Java expansion service contained Python specific PickleCoder that Java SDK cannot interpret. For example, this could be due to following.

• Input PCollection that is fed into the cross-language transform in Python side uses a Python specific type. In this case, the PTranform that produced the input PCollection has to be updated to produce outputs that use Beam's Standard Coders to make sure Java side can interpret such coders.
• Input PCollection uses standard coders but Python type inferencing results in picking up the PickleCoder. This can usually be resolved by annotating the predecessor Python transform with the correct type annotation using the with_output_types tag. See this Jira for an example.

How to set Java log level from a Python pipeline that uses Java transforms

...