[feature] Instrument goProbe with metrics

[els0r]

Closes #173

[fako1024]

Would appreciate if you could take a look at the feedback. Aside from that I've observed that when enabling profiling and metrics in the config:

   "api": {
     "addr": "127.0.0.1:3005",
     "metrics": true,
     "profiling": true,
   },

and I then try to pull a CPU profile (no matter if via the browser or via go tool pprof) I always get an empty profile (no error, just no samples; heap etc. work just fine). Can you confirm this? Am I missing something here?

[els0r]

Would appreciate if you could take a look at the feedback. Aside from that I've observed that when enabling profiling and metrics in the config:

   "api": {
     "addr": "127.0.0.1:3005",
     "metrics": true,
     "profiling": true,
   },

and I then try to pull a CPU profile (no matter if via the browser or via go tool pprof) I always get an empty profile (no error, just no samples; heap etc. work just fine). Can you confirm this? Am I missing something here?

I'll double check with your configuration. But I specifically used a /pprof endpoint to test an API related metric, so not sure what's up in your case.

The path is /debug/pprof/profile for a CPU profile. Did you use that?

[fako1024]

Would appreciate if you could take a look at the feedback. Aside from that I've observed that when enabling profiling and metrics in the config:

   "api": {
     "addr": "127.0.0.1:3005",
     "metrics": true,
     "profiling": true,
   },

and I then try to pull a CPU profile (no matter if via the browser or via go tool pprof) I always get an empty profile (no error, just no samples; heap etc. work just fine). Can you confirm this? Am I missing something here?

I'll double check with your configuration. But I specifically used a /pprof endpoint to test an API related metric, so not sure what's up in your case.

The path is /debug/pprof/profile for a CPU profile. Did you use that?

Jup, I used this command line call (on top of just navigating via the browser):

go tool pprof http://127.0.0.1:3005/debug/pprof/profile

And it definitively triggers the profile and pulls it (takes 30s by default, and I can set the time using the seconds query parameter), it's just always empty...

[els0r]

Jup, I used this command line call (on top of just navigating via the browser):

go tool pprof http://127.0.0.1:3005/debug/pprof/profile

And it definitively triggers the profile and pulls it (takes 30s by default, and I can set the time using the seconds query parameter), it's just always empty...

Could you please run

curl  http://127.0.0.1:3005/debug/pprof/profile\?seconds=300 -o cpu.profile

and see if it still occurs? I was also getting empty profiles for 30 seconds (using curl), while the profile for 300s was there, but sparse. I ran it on a node with little traffic, so this may just be a case of insufficient sampling.

[els0r]

@fako1024 : feedback amended. Ready when you are.

[fako1024]

Thanks for incorporating the feedback, LGTM!

[fako1024]

Jup, I used this command line call (on top of just navigating via the browser):

go tool pprof http://127.0.0.1:3005/debug/pprof/profile

And it definitively triggers the profile and pulls it (takes 30s by default, and I can set the time using the seconds query parameter), it's just always empty...

Could you please run

curl  http://127.0.0.1:3005/debug/pprof/profile\?seconds=300 -o cpu.profile

and see if it still occurs? I was also getting empty profiles for 30 seconds (using curl), while the profile for 300s was there, but sparse. I ran it on a node with little traffic, so this may just be a case of insufficient sampling.

Interesting - indeed after 300s I get something:

Duration: 300.01s, Total samples = 140ms (0.047%)

However, that's quite an aggressive sampling rate. There's also this issue gin-contrib/pprof#10 (which was closed with a hint at sampling). Given the fact that a "normal" profile run with StartCPUProfile() yields a much better sampling rate I think we need to add a call to runtime.SetCPUProfileRate(hz) (because that's what is being done in StartCPUProfile()): https://cs.opensource.google/go/go/+/refs/tags/go1.21.0:src/runtime/pprof/pprof.go;l=771

Belay that, the pprof wrapper calls StartCPUProfile(), so apparently I was just being dumb here. All good.