...

How to set Java log level from a Python pipeline that uses Java transforms

For supported runners (e.g. portable runners and Dataflow runner), you can set the log level of Java transforms in the same way of setting python module log level overrides, specifically, using the --sdk_harness_log_level_overrides pipeline option. The python_underline_style option names will be automatically translated to Java smallCamel style and recognized by the Java SDK harness.

If the runner does not support the automatic mapping of options, One can try adding the corresponding pipeline option as a local pipeline option explicitly in Python side. For example, to suppress all logs from Java package org.apache.kafka package you can do following.

1. Add a Python PipelineOption that represents the corresponding Java PipelineOption available here. This can be simply added to your Python program that starts up the Beam job.

   
   

   Code Block
   language py
   class JavaLoggingOptions(PipelineOptions):   @classmethod   def _add_argparse_args(cls, parser):     parser.add_argument(         '--sdkHarnessLogLevelOverrides',         default={},         type=json.loads,         help=(           'Java log level overrides'))

   
   

2. Specify the additional PipelineOption as a parameter when running the Beam pipeline.

   Code Block
   language bash
   --sdkHarnessLogLevelOverrides "{\"org.apache.kafka\":\"ERROR\"}"

...

Debugging a Python Test that calls a Java transform

...