Page History

...

• Follow the instructions in Go SDK Developer README at the sdks/go root for getting started with working on the Go SDK, in particular with how the Beam repo needs to be nested in the GOPATH.
• Failure to do so will mean you won't be executing your changes to the SDK.
Use the Python Portable Runner for end to end testing and validation of pipelines
• Follow the instructions in Python Tips to have a working python SDK environment.
• Ensure you're in the python virtual env.
• From the sdks/python directory, start a stand alone Python Portable Runner Job Service  
  • python -m apache_beam.runners.portability.local_job_service_main --port=8099
• Submit jobs with the flags :

  •  --runner=universal --endpoint=localhost:8099 --environment_type=LOOPBACK

• If this seems like too many steps, consider making the Python Portable runner more easily accessible to the Go SDK – BEAM-11077
• Consider alternatively using a local Spark or Flink instance to test pipelines. They can be spun up via beam repo gradle commands:
  • Flink ./gradlew :runners:flink:1.13:job-server:runShadow 
  • Spark ./gradlew :runners:spark:3:job-server:runShadow 
• Do not use the "Go Direct" runner. It's not much more than a toy suitable for limited basic batch testing. It's a separate task to make it more generally useful – – BEAM-11076

Testing

ValidatesRunner Tests

...