...

• s.apache.org/beam-community-metrics
• Design docs:
  

  • Beam Fast Precommits

  • Increase Beam post-commit tests stability

  • Improving Beam Code Review

  • Beam Community Metrics Infrastructure

• Source code: .test-infra/metrics
• Feature Requests: JIRA label community-metrics
• Monitoring: Jenkins beam_Prober_CommunityMetrics

Updating and Deploying

Instructions for developing and making updates to the metrics pipelines and dashboards are checked-in: README.md.

...

1. Backup the database. Login to the Database Backups page for beammetrics and create a backup in case anything goes wrong.
   
   
2. Connect to the database. From a console, login using gcloud command:
   gcloud beta sql beta connect beammetrics --database=beammetrics beammetrics --user=<your_username>


3. Verify the data to migrate. Craft a SELECT query for the data you plan to update. For example:
   

   SELECT *
FROM jenkins_builds jb
WHERE jb.job_name = 'beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle'
  AND NOT EXISTS (
    SELECT *
    FROM jenkins_builds
    WHERE job_name = 'beam_PostCommit_Java_ValidatesRunner_Gearpump'
      AND build_id=jb.build_id
  );



   Note that the import scripts will import recent history with the migrated job name from Jenkins; the NOT EXISTS clause above ensures that the duplicate history is not migrated from the previous name (it will be removed later). The UPDATE command will fail without this condition.
   
   
4. Update the data. After validating the query targets the intended rows, modify it to update the necessary fields:
   
   UPDATE jenkins_builds jb
SET job_name = 'beam_PostCommit_Java_ValidatesRunner_Gearpump'
WHERE <query-from-above>;


5. Remove duplicate history. The import script will import recent history with the migrated job name from Jenkins. The final step is to remove the redundant history with the old job name:
   
   DELETE jenkins_builds
WHERE job_name = 'beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle'

...

1. Log-in at http://localhost:3000 with username admin and the value specified for GF_SECURITY_ADMIN_PASSWORD in docker-compose.yml.
2. Add Postgres as a data source:
   1. Click the 'Add data source' button.
   2. Fill out the following config:
      • Name: BeamPSQL
      • Type: PostgreSQL
      • Host beampostgresql:5432
      • Database: beam_metrics
      • User: admin
      • Password: POSTGRES_PASSWORD in docker-compose.yml.
      • SSL Mode: Disable
3. Change default organization name to "Beam" to let anonymous users browse dashboards without logging in. This must be done manually, since it's impossible to do it via configuration (see issue: https://github.com/grafana/grafana/issues/2908).
   1. Log in as an administator
   2. Go to 'Server Admin' / 'Orgs'
   3. There should be one organization called 'Main Org.' Click the name and change it to "Beam"
4. Change home dashboard. Just like in the previous step, this must be done manually, due to https://github.com/grafana/grafana/issues/10266.
   1. Log in as an administrator
   2. Click at top-left on "Home". Find a dashboard called "Home / Getting Started" and mark it as favourite (give it a star)
   3. Go to 'Configuration' / 'Preferences'
   4. Choose "Home / Getting Started" from a drop-down list next to the 'Home Dashboard' label

To configure the InfluxDB Data Source:

1. Log-in at http://localhost:3000 as specified above.
2. Add InfluxDB as a data source:
   1. Click the 'Add data source' button
   2. Fill out the following config:
      • Name: BeamInfluxDB
      • URL: http://localhost:8086
      • Access: Browser
      • Database: beam_test_metrics

Updating Dashboards

The Grafana dashboards are exported as JSON files in the codebase. Dashboards can be easily exported from the UI.

...