Skip to content

Commit

Permalink
server: enable continuous CPU profiler
Browse files Browse the repository at this point in the history
This patch enables the CPU profile with a threshold of 75% and a max
frequency of 20min.

The motivation for this change is that recently, I have noticed a few
cases during investigations where CPU utilization spikes but we lack
profiles after the fact. This hinders our ability to dig deeper into the
source of high CPU usage. Having profiles can help inform us of future
AC integrations that we may need, or other performance improvements we
can do elsewhere.

Informs #97699.

Release note (ops change): CRDB will now automatically generate CPU
profiles if there is an increase in CPU utilization. This can help
investigate possible issues after the fact.
  • Loading branch information
aadityasondhi committed Feb 8, 2024
1 parent 17f1eac commit 9036430
Showing 1 changed file with 6 additions and 4 deletions.
10 changes: 6 additions & 4 deletions pkg/server/profiler/cpuprofiler.go
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,6 @@ package profiler

import (
"context"
"math"
"os"
"runtime/pprof"
"time"
Expand Down Expand Up @@ -41,7 +40,8 @@ var cpuUsageCombined = settings.RegisterIntSetting(
"the profiler will never take a profile and conversely, if a value"+
"of 0 is set, a profile will be taken every time the cpu profile"+
"interval has passed or the provided usage is increasing",
math.MaxInt64,
65,
settings.PositiveInt,
)

var cpuProfileInterval = settings.RegisterDurationSetting(
Expand All @@ -51,14 +51,16 @@ var cpuProfileInterval = settings.RegisterDurationSetting(
// account the high water mark seen. Without this, if CPU ever reaches 100%,
// we'll never take another profile.
"duration after which the high water mark resets and a new cpu profile can be taken",
5*time.Minute, settings.PositiveDuration,
20*time.Minute,
settings.PositiveDuration,
)

var cpuProfileDuration = settings.RegisterDurationSetting(
settings.ApplicationLevel,
"server.cpu_profile.duration",
"the duration for how long a cpu profile is taken",
10*time.Second, settings.PositiveDuration,
10*time.Second,
settings.PositiveDuration,
)

const cpuProfFileNamePrefix = "cpuprof"
Expand Down

0 comments on commit 9036430

Please sign in to comment.