Page History

Sketch a working skeleton of portable Kafka Streams Runner for Apache Beam. The runner should be able to run basic portable pipelines and be a baseline implementation for further development, feature additions and performance optimization.

A more detailed design document shall be attached to the github tracking issue.

The Beam project has a few examples where hardware accelerators can be used to run models. See https://github.com/apache/beam/blob/master/examples/notebooks/beam-ml/dataflow_tpu_examples.ipynb

This project is to improve on the available set of examples by building starter examples that allow a user to write code that slowly builds up to using these hardware accelerators. The idea would be:

• A simple python script that runs slowly without HW accelerators
• A script that shows improvements when using them
• A training job that uses accelerators
• A Beam pipeline that can train multiple models in parallel using accelerators
• A blog post that can serve as a guide for anyone learning to use hardware accelerators

These would run continuously to ensure their freshness.

This project consists in a series of tasks that build a sort of 'infra platform' for Beam. Some tasks include:

• Automated cleaning of infrastructure: [Task]: Build a cleaner for assets in the GCP test environment #33644
• Implement Infra-as-code for Beam infrastructure
  • Implement access permissions using IaC: [Task]: Build a cleaner for assets in the GCP test environment #33644
  • Implement drift detection for IaC resources for Beam
  • Implement 'best-practice' key management for Beam (i.e. force key rotation for service account keys, and store in secret manager secrets)

A quality proposal will include a series of features beyond the ones listed above. Some ideas:

• Detection of policy breakages, and nagging to fix
• Security detections based on cloud logging
• others?