Page History

...

• Follow the instructions in Go SDK Developer README at the sdks/go root
• Use the Python Portable Runner for end to end testing and validation of pipelines
  • Follow the instructions in Python Tips to have a working python SDK environment.
  • Ensure you're in the python virtual env.
  • From the sdks/python directory, start a stand alone Python Portable Runner Job Service  
    • python -m apache_beam.runners.portability.local_job_service_main --port=8099
  • Submit jobs with the flags :

    •  --runner=universal --endpoint=localhost:8099 --environment_type=LOOPBACK

  • If this seems like too many steps, consider making the Python Portable runner more easily accessible to the Go SDK – BEAM-11077
• Consider alternatively using a local Spark or Flink instance to test pipelines. They can be spun up via beam repo gradle commands:
  • Flink ./gradlew :runners:flink:1.1316:job-server:runShadow 
  • Spark ./gradlew :runners:spark:3:job-server:runShadow 
• Do not use the "Go Direct" runner. It's not much more than a toy suitable for limited basic batch testing. It's a separate task to make it more generally useful – BEAM-11076

...

1. All integration tests should be in a subpackage, not in the root integration package.
2. Integration tests should be executing pipelines with ptest.Run(). Tests that do not execute pipelines likely do not belong as integration tests.
   1. If this is a new package, it's required that TestMain(m *testing.M) is defined and calls ptest.Main(m).

3. The integration package defines a way to enable test filtering for your integration test. Call integration.CheckFilters(t) at the start of the test. integration.go contains the various runner specific filters, to filter out tests per runner, if they don't support the given feature under test. All integration tests should have test filtering enabled.

   1. The test filter will fail the test in the event TestMain doesn't call ptest.Main.

...