Add Prow job to run periodic performance benchmarks

[ivanvc]

What would you like to be added?

As part of #16467, create a Prow job that nightly runs the rw-benchmarks. Storing the results as an artifact so results can be processed later and have an idea of how the development is affecting the project's performance.

Why is this needed?

To start collecting performance benchmarks to establish a project perspective on performance.

[jmhbnz]

Added to our tracking under #16467.

[ivanvc]

The first five performance jobs are running in the K8s test EKS clusters. We still need to add the last three jobs.

[serathius]

Thanks for integrating benchmarks into the prow, however I was hoping that we will do the work in semi order described in #16467 (comment), so before we start running periodic benchmark we agree on what to measure first.

Having the rw-benchmark running on prow is good, however I would want to avoid a case where we later decide that we need different approach and throw out your work.

[ivanvc]

This makes sense, @serathius; thanks for your input. @jmhbnz, should we regroup and formalize a proposal?

[jmhbnz]

This makes sense, @serathius; thanks for your input. @jmhbnz, should we regroup and formalize a proposal?

Certainly. I am currently taking some time off to move house but will be returning to open source work 6th May and very keen to regroup when I return. Feel free to update #16467 with any thoughts in the interim and thanks again for your work getting these initial tests of prow for heatmaps running @ivanvc