Codespeed / Benchmarks

Jenkins running benchmarks: http://codespeed.dak8s.net:8080

Benchmarks WebUI: http://codespeed.dak8s.net:8000

If you merge a performance critical change (e.g. code paths executed per record, state entry etc) to master, you should verify that it did not cause a regression comparing to the previous state of the master. There is an existing performance test suite which periodically runs on the master branch. You can check the timeline of its results after some time in this UI. If you are unsure about your changes and want to check the regression before merging, contact a PMC to submit a benchmark request and check results in the comparison UI.

Check also more details in the mailing list announcement.

If you want to know how to execute the benchmarks locally, please take a look at the benchmarks' readme. 

Submitting a benchmark request

In order to submit a benchmark request you need to contact someone with write access to https://github.com/dataArtisans/flink/tree/benchmark-request (benchmarking infrastructure is hosted using resources outside of the Apache foundation). If you already have the write access you should:

1. Push your changes to https://github.com/dataArtisans/flink/tree/benchmark-request branch, probably using `–force` to overwrite previous changes. In this step, please make sure that nobody else is doing some benchmarking at the same time. You can check this by looking at the last modification date of this branch and or when was the last benchmark request run on the Jenkins. If last activity was more than a day ago, you are probably good to go. If not, please contact the person that made the most recent changes.
2. Trigger a new build on Jenkins benchmark request project.
3. Once benchmark request build finishes, the results will be:
   • archived as the build artefact on the Jenkins as CSV file   
   • pushed to comparison UI

Jenkins

The general structure of projects on Jenkins is as follows.

Main benchmarks are executed in project `flink-master-benchmarks`. This job periodically executes main benchmarks and uploads them to the WebUI. `flink-statebackend-benchmark` is doing the same thing, but for the State Backend benchmarks.

Each of those two projects, have two “benchmark-request” projects (`flink-benchmark-request` and `flink-statebackend-benchmark-request` respectively). They execute the same set of benchmarks (again main and State Backend respectively), but they are not testing code from the master branch, but from a different branch (usually benchmark-request), and upload the results to the WebUI as comparison against the latest master results. Benchmark request jobs have to be triggered manually on Jenkins.