Add some metrics to dispute-coordinator.

[eskimor]

It is still quite slow apparently. For now to be deployed on Versi - not sure we want to keep them yet.

[sandreim]

We have quite a lot of metrics, and most of them are not really used when debugging. What do you think if we take a similar approach to logs, such that we can group them in sets, say debug metrics and prod metrics that we can enable via CLI ?

[eskimor]

Not sure it is worth the trouble, but an interesting idea!

[sandreim]

The main benefit would be lower scraping time and reducing the prometheus load which is a problem for large deployments.

[eskimor]

Yep - I would lean to just removing those metrics and adding them back on a per need basis. I have the feeling it is not worth keeping them around. Creating a PR like this one is rather cheap and you have the added benefit that you can add precisely what you need at the moment - no matter how ugly, as it is just for debugging.

[eskimor]

Ok requests are not costly nor frequent - I really cannot make any sense of those ToF values. Also when looking at logs: Everything is blazingly fast now - together with requests that trickle in once every few seconds. I am getting the feeling that something is off with that ToF metric, I am also not seeing any consequences: With an occasional ToF >0.5 seconds, the provisioner should actually time out - never happens as far as I can see.

[eskimor]

What does add up a little bit is importing those backing votes.