...

--sdkHarnessLogLevelOverrides "{\"org.apache.kafka\":\"ERROR\"}"

Debugging a Python Test that calls a Java transform

The public documentation is great on creating and using cross-language transforms. For developers, however, running tests and debugging them is a bit different.  

We will showcase how to debug the following test sql_test.SqlTransformTest.test_two_pcoll_same_schema.

Set up the input arguments to the Python test.

Beside the test function, right click the play button, and click "Modify Run Configuration..."

Image Added

In Additional Arguments , add the arguments. For this example, we add 

--beam_services='{\":sdks:java:extensions:sql:expansion-service:shadowJar\": \"localhost:8097\"}'"

Image Added

Run the Java expansion service

The Gradle build file defines how to run the expansion service. To do this manually, go to the Gradle tab in Intellij, go to this path beam/sdks/java/extensions/sql/expanstion-service/tasks/other/ , and find the `runExpansionService` entry.

• To run the expansion service without debugging, right click runExpansionService , and simply click "run".
• If you want to debug the Java code, set breakpoints in the IDE in Java files that are part of the expansion service. (i.e. ExpansionService.Java). Then click "Debug"

Image Added

Run the Python test

• If you don't want to debug the Python test, right click the button next to the test, and click Run 'pytest for sql_test...' 
• If you do want to debug the Python test, first set the breakpoints in your code. Then right click the button next to the test, and click 'Debug 'pytest for sql_test...' 

Notes

You can choose to run only Python in debug mode and the expansion service not in debug mode; Java expansion service in debug mode and the Python code not in debug mode; both in debug mode; or neither in debug mode.