Set up a pprof server with optional mutex profiling in cmdutil #216

sherryyaosf · 2024-07-26T06:05:31Z

Description

We want to configure the pprof listener that is capable of running mutex profiles on server set-up in cmdutil/debug. This PR created a new NewPprofServer method to the cmdutil/debug package, allowing for the setup of a pprof server with configurable profiling types and optional mutex profiling.

Changes

Add PProfServerConfig struct: holds configuration for the pprof server.
Add PProfServer struct: wraps a pprof server
Implement NewPProfServer function: creates and configures the pprof server.
Implement Run and Stop methods: the NewPProfServer function includes Run and Stop methods for managing the pprof server lifecycle.
Write unit tests for NewPProfServer

Metadata
W-16252388

slizco · 2024-08-01T19:56:54Z

cmdutil/server.go

+		config.Addr = "127.0.0.1:9998" // Default port
+	}
+
+	return NewContextServer(func(ctx context.Context) error {


I think we should return a *Server type and then we can build a context server if needed. That follows the debug.Server pattern:
https://github.com/heroku/x/blob/master/cmdutil/debug/debug.go#L48-L52

slizco · 2024-08-01T19:57:11Z

cmdutil/server.go

@@ -93,3 +97,80 @@ func MultiServer(servers ...Server) Server {
 		StopFunc: s.Stop,
 	}
 }
+
+// NewPprofServer sets up a pprof server with optional mutex profiling.


Why have this commented out code? Can we just remove it?

slizco · 2024-08-01T19:57:37Z

cmdutil/server.go

@@ -4,6 +4,10 @@ package cmdutil

 import (
 	"context"
+	"net/http"


Let's put the pprof server in the debug package. It is used for debugging

slizco · 2024-08-01T19:59:36Z

cmdutil/server.go

+		mux := http.NewServeMux()
+
+		profileHandlers := map[string]http.HandlerFunc{
+			"index":        pprof.Index,


I think the handlers should be configurable. We don't need all of these profile types most of the time.

slizco · 2024-08-01T20:00:45Z

cmdutil/server.go

+			pprofSrv.Close()
+		}()
+
+		return pprofSrv.ListenAndServe()


Let's have some logging when we start the server, including what port we are binding to: https://github.com/heroku/x/blob/master/cmdutil/debug/debug.go#L48-L52

slizco

I'd also like to see a unit test of this

slizco · 2024-08-02T17:14:11Z

cmdutil/debug/debug.go

+	logger      logrus.FieldLogger
+	addr        string
+	done        chan struct{}
+	pprofServer *http.Server


I'd like to see this as a separate server. So something like debu.PProfServer.

slizco · 2024-08-02T17:23:31Z

cmdutil/debug/debug.go

+	}
+
+	l.WithFields(logrus.Fields{
+		"at":      "binding",


Binding log should happen when Run() is called. The server does not actually start until Run()

slizco · 2024-08-02T17:23:54Z

cmdutil/debug/debug.go

+	for profile, handler := range config.ProfileHandlers {
+		if handler != nil {
+			if profile == "mutex" {
+				runtime.SetMutexProfileFraction(2)


I think this should be configurable on the profile config.

I mean something like this:

type PProfServerConfig struct { MutextProfileFraction int }

If set to zero and the mutex profile is enabled, we can use a reasonable default of 2.

slizco · 2024-08-02T17:24:33Z

cmdutil/debug/debug.go

+				"service": "pprof",
+				"profile": profile,
+			}).Info("Added pprof profile handler")
+		} else {


We don't need this else. If someone has passed in a nil-value handler, that is poor configuration we should not support. A nil-pointer exception and subsequent panic are acceptable

slizco · 2024-08-02T17:27:02Z

cmdutil/debug/debug.go

+				"at":      "adding",
+				"service": "pprof",
+				"profile": profile,
+			}).Info("Added pprof profile handler")


Don't think we need to log this. If we want, we can do a simple strings.Join() on the profiles slice and then log profiles="heap,cpu,blah..." when we log the port binding in Run().

slizco · 2024-08-05T18:13:42Z

cmdutil/debug/debug.go

+}
+
+// ProfileConfig holds the configuration for the pprof server.
+type ProfileConfig struct {


Since this is used to configure a PProfServer, let's call it PProfServerConfig.

slizco · 2024-08-05T18:15:15Z

cmdutil/debug/debug.go

+	for profile, handler := range config.ProfileHandlers {
+		if handler != nil {
+			if profile == "mutex" {
+				runtime.SetMutexProfileFraction(2)


I mean something like this:

type PProfServerConfig struct { MutextProfileFraction int }

If set to zero and the mutex profile is enabled, we can use a reasonable default of 2.

slizco · 2024-08-05T18:16:08Z

cmdutil/debug/debug.go

+	if s.pprofServer != nil {
+		go func() {
+			if err := s.pprofServer.ListenAndServe(); err != nil && err != http.ErrServerClosed {
+				s.logger.WithError(err).Error("pprof server error")


We'll want the error to be returned by Run()

slizco · 2024-08-05T18:22:54Z

cmdutil/debug/debug.go

+			if profile == "mutex" {
+				runtime.SetMutexProfileFraction(2)
+			}
+			mux.HandleFunc("/debug/pprof/"+profile, handler)


We don't need to raw construction of the handlers like this. The pprof.Handler function can construct the handlers for us: https://pkg.go.dev/net/http/pprof#Handler

…return

slizco · 2024-08-05T22:18:03Z

cmdutil/debug/debug.go

+			}
+			runtime.SetMutexProfileFraction(config.MutexProfileFraction)
+		}
+		mux.Handle("/debug/pprof/"+profile, pprof.Handler(profile))


Now I am thinking that we should just use pprof.Index and generate a handler like so:

http.HandlerFunc(pprof.Index)

Then you can avoid using mux and having to pass in the list of profiles.

slizco · 2024-08-05T22:20:55Z

cmdutil/debug/debug.go

+	for _, profile := range config.ProfileNames {
+		if profile == "mutex" {
+			if config.MutexProfileFraction == 0 {
+				config.MutexProfileFraction = 2 // Use default value of 2 if not set


Let's not modify the config. That is unexpected behavior. Instead, we can use a local variable:

mpf := defaultMutextProfileFraction // value is 2 if config.MutexProfileFraction != 0 { mpf = config.MutextProfileFraction } runtime.SetMutextProfileFraction(mpf)

slizco

Some minor things but this looks good!

slizco · 2024-08-06T19:20:08Z

cmdutil/debug/debug.go

+		"addr":    s.addr,
+	}).Info()
+
+	if s.pprofServer != nil {


If it is nil, I think we can return an error.

slizco · 2024-08-06T19:36:16Z

cmdutil/debug/debug_test.go

+			}
+			runtime.SetMutexProfileFraction(tt.expectedMutexFraction) // Reset to the expected value
+
+			urls := []string{


I think verifying the paths is probably overkill since that is functionality within pprof package.

Sherry Yao added 2 commits July 25, 2024 22:57

Set up a pprof server with optional mutex profiling in cmdutil

ebb724d

Set up a ReadHeaderTimeout

bd669c7

slizco requested review from agnaite, cgfuh, archa347 and Vamsi-Mundra August 1, 2024 19:51

slizco requested changes Aug 1, 2024

View reviewed changes

Put the method in debug package and improve it

850c21a

sherryyaosf requested a review from slizco August 2, 2024 07:32

slizco requested changes Aug 2, 2024

View reviewed changes

Sherry Yao added 3 commits August 2, 2024 11:57

Make PProfServer a seperate server and update the logging info

3f6f3f6

Remove unnecessary changes

9b39c0d

Run golangci-lint

51a1892

sherryyaosf requested a review from slizco August 2, 2024 19:47

slizco reviewed Aug 5, 2024

View reviewed changes

Updated PProfServerConfig, used pprof.Handler and included Run() err …

1a67670

…return

sherryyaosf requested a review from slizco August 5, 2024 19:57

slizco reviewed Aug 5, 2024

View reviewed changes

Sherry Yao added 3 commits August 5, 2024 16:03

Updated mutex and used pprof.Index

a7b309c

Run unit test

37fed8c

Update unit test

ad16243

sherryyaosf requested a review from slizco August 6, 2024 18:51

slizco approved these changes Aug 6, 2024

View reviewed changes

Considered pprofServer as nil and simplified test path for HTTP requests

60faef3

sherryyaosf merged commit 1e0ca7c into master Aug 6, 2024
3 checks passed

sherryyaosf deleted the sherryyao-branch branch August 6, 2024 21:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Set up a pprof server with optional mutex profiling in cmdutil #216

Set up a pprof server with optional mutex profiling in cmdutil #216

sherryyaosf commented Jul 26, 2024 •

edited

Loading

slizco Aug 1, 2024

slizco Aug 1, 2024

slizco Aug 1, 2024

slizco Aug 1, 2024

slizco Aug 1, 2024

slizco left a comment

slizco Aug 2, 2024

slizco Aug 2, 2024

slizco Aug 2, 2024

slizco Aug 5, 2024

slizco Aug 2, 2024

slizco Aug 2, 2024

slizco Aug 5, 2024

slizco Aug 5, 2024

slizco Aug 5, 2024

slizco Aug 5, 2024

slizco Aug 5, 2024

slizco Aug 5, 2024

slizco left a comment

slizco Aug 6, 2024

slizco Aug 6, 2024

Set up a pprof server with optional mutex profiling in cmdutil #216

Set up a pprof server with optional mutex profiling in cmdutil #216

Conversation

sherryyaosf commented Jul 26, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

slizco left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

slizco left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sherryyaosf commented Jul 26, 2024 •

edited

Loading