Add GRPC client and server metrics

[aliaqel-stripe]

This PR adds GRPC Client and Server interceptors that emit GRPC request metrics from the perspective of the Client and Server. This allows developers to observe the performance of the adapter -> operator grpc connection.

To implement this, I use the upstream grpc-middleware with default prometheus metrics. I create helpers to create the metrics objects in metricscollector and then add them to the client/server if prometheus metrics are enabled.

Tested on cluster, screenshot from Grafana:

Checklist

• I have verified that my change is according to the deprecations & breaking changes policy
• Changelog has been updated and is aligned with our changelog requirements
• A PR is opened to update the documentation on (repo) (if applicable)
• Commits are signed with Developer Certificate of Origin (DCO - learn more)

Fixes #5502

Relates to kedacore/keda-docs#1340

[SpiritZhou]

Could you add some unit tests/integration tests for this feature?

[aliaqel-stripe]

Could you add some unit tests/integration tests for this feature?

Oh yes, for sure. I found the metrics tests, I will add checks for these there.

[aliaqel-stripe]

Could you add some unit tests/integration tests for this feature?

@SpiritZhou Can the tests run in Kind? We don't use Helm on our clusters nor do we use kubernetes networking, so I have to run locally.

[JorTurFer]

@SpiritZhou Can the tests run in Kind? We don't use Helm on our clusters nor do we use kubernetes networking, so I have to run locally.

Hello!
Definitively you can! You can install KEDA in your local cluster using this guide. Then, you can execute the tests directly pointing to your local cluster. (If you want to execute OTEL metrics test too, you need to deploy also the collector)

[JorTurFer]

/run-e2e prometheus_metrics
Update: You can check the progress here

[aliaqel-stripe]

@JorTurFer can you rerun the prom tests? I set the family name incorrectly in the tests and that's why the histogram tests failed.

[aliaqel-stripe]

this guide

lol our security tools won't let me run mockgen outside of the dev container on my laptop lol

@JorTurFer can you rerun the prometheus e2e tests?

[JorTurFer]

/run-e2e prometheus_metrics
Update: You can check the progress here

[aliaqel-stripe]

I ended up removing the test for the histogram, there are three metrics in that family and that's causing the crash so I'm just going to assume if the other 4 metrics are there it's there as this is upstream metrics functionality and not something we're reporting to our selves.

[aliaqel-stripe]

@JorTurFer lol, can you run the e2e tests one more time :)

[JorTurFer]

/run-e2e prometheus_metrics
Update: You can check the progress here

[aliaqel-stripe]

@JorTurFer Can you run e2e on these new changes?

[zroubalik]

/run-e2e prometheus_metrics
Update: You can check the progress here

[zroubalik]

This is a great addition. My only concern is about naming of these metrics. The grpc there is too generic. Maybe keda_metricservice_ or something around that?

[aliaqel-stripe]

This is a great addition. My only concern is about naming of these metrics. The grpc there is too generic. Maybe keda_metricservice_ or something around that?

I can make that change.

[zroubalik]

This is a great addition. My only concern is about naming of these metrics. The grpc there is too generic. Maybe keda_metricservice_ or something around that?

I can make that change.

Omg, I am very sorry, the prefix should be keda_internal_metricsservice_, sorry forgot about that.

[aliaqel-stripe]

This is a great addition. My only concern is about naming of these metrics. The grpc there is too generic. Maybe keda_metricservice_ or something around that?

I can make that change.

Omg, I am very sorry, the prefix should be keda_internal_metricsservice_, sorry forgot about that.

Haha all good, will fix it quickly!

[aliaqel-stripe]

This is a great addition. My only concern is about naming of these metrics. The grpc there is too generic. Maybe keda_metricservice_ or something around that?

I can make that change.

Omg, I am very sorry, the prefix should be keda_internal_metricsservice_, sorry forgot about that.

Updated the name! Can you re-run e2e tests when they're ready?

[aliaqel-stripe]

@zroubalik I can't see the snyk test failure because I don't have access to their UI

[zroubalik]

/run-e2e prometheus_metrics
Update: You can check the progress here

[zroubalik]

@zroubalik I can't see the snyk test failure because I don't have access to their UI

that's fine