[Profiler] Add `timer_create`-based CPU profiling on Linux #5476

gleocadie · 2024-04-19T18:47:24Z

Summary of changes

Add timer_create-based CPU profiling on linux.

Reason for change

On Linux, at each profiling tick, the sampling thread goes over the thread proc information (reading files) to 1) check if it's currently running on a CPU 2) retrieve the CPU time.

This is a significant overhead ( in addition to the stackwalking) and has a greater impact (relatively) in case of job application (applications that wait most of the time, and time to time, process requests/jobs)

With this (and the other improvements we've done in the past), this decrease the profiler overhead by at least 2x (2.46, the profiler had a 2.5s CPU overhead over a minute, with the various improvements, its goes to 1s CPU over a minute).

Implementation details

Revisit the ProfilerSignalManager to be able to get a manager by signal
Add mmap-based memory resource
Allow to change CPU profiler type ( manual or timer_create)
Allow to change the CPU sampling interval

Test coverage

Add unit tests for new classes and new configuration
Add integration tests for the new CPU profiler
Add benchmark tests
Add throughput test

Other details

datadog-ddstaging · 2024-04-19T19:19:47Z

Datadog Report

Branch report: gleocadie/add-timer-create-based-cpu-profiling
Commit report: 46805c1
Test service: dd-trace-dotnet

✅ 0 Failed, 333689 Passed, 1801 Skipped, 43m 30.98s Wall Time
⌛ 1 Performance Regression

⌛ Performance Regressions vs Default Branch (1)

Profiler_exceptions - scenarios 4.71s (+374.94ms, +9%) - Details

andrewlock · 2024-04-19T19:54:44Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5476) - mean (71ms)  : 64, 77
     .   : milestone, 71,
    master - mean (74ms)  : 61, 87
     .   : milestone, 74,

    section CallTarget+Inlining+NGEN
    This PR (5476) - mean (1,018ms)  : 994, 1041
     .   : milestone, 1018,
    master - mean (1,018ms)  : 999, 1036
     .   : milestone, 1018,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5476) - mean (108ms)  : 106, 111
     .   : milestone, 108,
    master - mean (109ms)  : 106, 111
     .   : milestone, 109,

    section CallTarget+Inlining+NGEN
    This PR (5476) - mean (708ms)  : 681, 734
     .   : milestone, 708,
    master - mean (715ms)  : 687, 744
     .   : milestone, 715,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5476) - mean (92ms)  : 89, 95
     .   : milestone, 92,
    master - mean (92ms)  : 90, 95
     .   : milestone, 92,

    section CallTarget+Inlining+NGEN
    This PR (5476) - mean (663ms)  : 633, 692
     .   : milestone, 663,
    master - mean (661ms)  : 638, 684
     .   : milestone, 661,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5476) - mean (190ms)  : 187, 193
     .   : milestone, 190,
    master - mean (191ms)  : 188, 194
     .   : milestone, 191,

    section CallTarget+Inlining+NGEN
    This PR (5476) - mean (1,108ms)  : 1085, 1130
     .   : milestone, 1108,
    master - mean (1,108ms)  : 1085, 1131
     .   : milestone, 1108,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5476) - mean (277ms)  : 272, 282
     .   : milestone, 277,
    master - mean (275ms)  : 270, 280
     .   : milestone, 275,

    section CallTarget+Inlining+NGEN
    This PR (5476) - mean (881ms)  : 855, 907
     .   : milestone, 881,
    master - mean (880ms)  : 859, 900
     .   : milestone, 880,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5476) - mean (265ms)  : 261, 269
     .   : milestone, 265,
    master - mean (265ms)  : 261, 269
     .   : milestone, 265,

    section CallTarget+Inlining+NGEN
    This PR (5476) - mean (865ms)  : 840, 891
     .   : milestone, 865,
    master - mean (859ms)  : 836, 882
     .   : milestone, 859,

andrewlock · 2024-04-19T21:44:11Z

Benchmarks Report for tracer 🐌

Benchmarks for #5476 compared to master:

1 benchmarks are slower, with geometric mean 1.127
All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	7.64μs	43.8ns	350ns	0.0185	0.00742	0	5.43 KB
master	`StartStopWithChild`	netcoreapp3.1	9.77μs	52.7ns	279ns	0.0143	0.00478	0	5.61 KB
master	`StartStopWithChild`	net472	16.2μs	44ns	170ns	1.04	0.327	0.0956	6.07 KB
#5476	`StartStopWithChild`	net6.0	7.74μs	43.9ns	320ns	0.0154	0.00384	0	5.43 KB
#5476	`StartStopWithChild`	netcoreapp3.1	9.89μs	55.6ns	361ns	0.0194	0.00968	0	5.62 KB
#5476	`StartStopWithChild`	net472	16.3μs	66.7ns	240ns	1.02	0.305	0.0882	6.08 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	461μs	318ns	1.23μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	615μs	154ns	556ns	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	829μs	454ns	1.76μs	0.414	3.3 KB
#5476	`WriteAndFlushEnrichedTraces`	net6.0	487μs	142ns	551ns	0	2.7 KB
#5476	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	643μs	303ns	1.14μs	0	2.7 KB
#5476	`WriteAndFlushEnrichedTraces`	net472	821μs	65.1ns	226ns	0.411	3.3 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net6.0	170μs	140ns	543ns	0.171	18.45 KB
master	`SendRequest`	netcoreapp3.1	192μs	115ns	415ns	0.191	20.61 KB
master	`SendRequest`	net472	0.000655ns	0.000313ns	0.00117ns	0	0 b
#5476	`SendRequest`	net6.0	170μs	162ns	606ns	0.255	18.45 KB
#5476	`SendRequest`	netcoreapp3.1	192μs	232ns	867ns	0.191	20.61 KB
#5476	`SendRequest`	net472	0.00128ns	0.000443ns	0.00172ns	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	539μs	1.65μs	6.37μs	0.551	0	0	41.83 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	657μs	390ns	1.4μs	0.331	0	0	41.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	845μs	3.46μs	12.9μs	8.28	2.48	0.414	53.22 KB
#5476	`WriteAndFlushEnrichedTraces`	net6.0	548μs	779ns	3.02μs	0.546	0	0	41.66 KB
#5476	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	644μs	1.19μs	4.46μs	0.327	0	0	41.66 KB
#5476	`WriteAndFlushEnrichedTraces`	net472	857μs	2.78μs	10.8μs	8.19	2.59	0.431	53.25 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteNonQuery`	net6.0	1.19μs	0.687ns	2.66ns	0.0143	1.02 KB
master	`ExecuteNonQuery`	netcoreapp3.1	1.7μs	1.86ns	7.21ns	0.0137	1.02 KB
master	`ExecuteNonQuery`	net472	2.02μs	1.1ns	3.98ns	0.156	987 B
#5476	`ExecuteNonQuery`	net6.0	1.26μs	0.612ns	2.37ns	0.0145	1.02 KB
#5476	`ExecuteNonQuery`	netcoreapp3.1	1.66μs	0.653ns	2.53ns	0.0134	1.02 KB
#5476	`ExecuteNonQuery`	net472	1.9μs	0.43ns	1.67ns	0.157	987 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`CallElasticsearch`	net6.0	1.22μs	0.891ns	3.45ns	0.0134	976 B
master	`CallElasticsearch`	netcoreapp3.1	1.49μs	2.98ns	11.5ns	0.0129	976 B
master	`CallElasticsearch`	net472	2.53μs	1.05ns	4.07ns	0.157	995 B
master	`CallElasticsearchAsync`	net6.0	1.34μs	0.7ns	2.71ns	0.0128	952 B
master	`CallElasticsearchAsync`	netcoreapp3.1	1.63μs	0.83ns	3.21ns	0.0137	1.02 KB
master	`CallElasticsearchAsync`	net472	2.6μs	2.55ns	9.86ns	0.167	1.05 KB
#5476	`CallElasticsearch`	net6.0	1.23μs	0.498ns	1.8ns	0.0137	976 B
#5476	`CallElasticsearch`	netcoreapp3.1	1.57μs	0.708ns	2.65ns	0.0133	976 B
#5476	`CallElasticsearch`	net472	2.56μs	0.613ns	2.37ns	0.157	995 B
#5476	`CallElasticsearchAsync`	net6.0	1.33μs	0.59ns	2.29ns	0.0133	952 B
#5476	`CallElasticsearchAsync`	netcoreapp3.1	1.65μs	0.742ns	2.77ns	0.0132	1.02 KB
#5476	`CallElasticsearchAsync`	net472	2.53μs	1.84ns	6.9ns	0.166	1.05 KB

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net6.0	1.19μs	0.861ns	3.33ns	0.0136	952 B
master	`ExecuteAsync`	netcoreapp3.1	1.57μs	1.13ns	4.23ns	0.0125	952 B
master	`ExecuteAsync`	net472	1.77μs	0.75ns	2.81ns	0.145	915 B
#5476	`ExecuteAsync`	net6.0	1.19μs	0.361ns	1.35ns	0.0131	952 B
#5476	`ExecuteAsync`	netcoreapp3.1	1.58μs	1.12ns	4.35ns	0.0127	952 B
#5476	`ExecuteAsync`	net472	1.74μs	1.11ns	4.29ns	0.145	915 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	4.18μs	2.87ns	11.1ns	0.0295	2.22 KB
master	`SendAsync`	netcoreapp3.1	5.12μs	2.8ns	10.8ns	0.0382	2.76 KB
master	`SendAsync`	net472	7.5μs	2.78ns	10.8ns	0.499	3.15 KB
#5476	`SendAsync`	net6.0	4.23μs	1.62ns	6.04ns	0.0296	2.22 KB
#5476	`SendAsync`	netcoreapp3.1	5.06μs	2.46ns	9.2ns	0.0379	2.76 KB
#5476	`SendAsync`	net472	7.67μs	3.75ns	14ns	0.496	3.15 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	1.56μs	0.561ns	2.1ns	0.0233	1.64 KB
master	`EnrichedLog`	netcoreapp3.1	2.06μs	0.433ns	1.56ns	0.0226	1.64 KB
master	`EnrichedLog`	net472	2.49μs	0.766ns	2.76ns	0.249	1.57 KB
#5476	`EnrichedLog`	net6.0	1.48μs	0.43ns	1.67ns	0.023	1.64 KB
#5476	`EnrichedLog`	netcoreapp3.1	2.21μs	1.08ns	4.03ns	0.0221	1.64 KB
#5476	`EnrichedLog`	net472	2.5μs	1.64ns	6.34ns	0.249	1.57 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net6.0	114μs	158ns	614ns	0.0563	0	4.28 KB
master	`EnrichedLog`	netcoreapp3.1	119μs	176ns	682ns	0	0	4.28 KB
master	`EnrichedLog`	net472	146μs	55.1ns	213ns	0.658	0.219	4.46 KB
#5476	`EnrichedLog`	net6.0	116μs	98ns	379ns	0.0566	0	4.28 KB
#5476	`EnrichedLog`	netcoreapp3.1	118μs	135ns	524ns	0.0589	0	4.28 KB
#5476	`EnrichedLog`	net472	147μs	52.1ns	202ns	0.659	0.22	4.46 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	3.06μs	1.54ns	5.75ns	0.0307	2.2 KB
master	`EnrichedLog`	netcoreapp3.1	4.4μs	1.28ns	4.81ns	0.0286	2.2 KB
master	`EnrichedLog`	net472	5μs	1.54ns	5.76ns	0.32	2.02 KB
#5476	`EnrichedLog`	net6.0	3.09μs	0.894ns	3.46ns	0.0293	2.2 KB
#5476	`EnrichedLog`	netcoreapp3.1	4.21μs	1.51ns	5.87ns	0.0294	2.2 KB
#5476	`EnrichedLog`	net472	4.85μs	1.77ns	6.84ns	0.32	2.02 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendReceive`	net6.0	1.38μs	2.34ns	8.77ns	0.0161	1.14 KB
master	`SendReceive`	netcoreapp3.1	1.72μs	2.91ns	11.3ns	0.0146	1.14 KB
master	`SendReceive`	net472	2.08μs	0.675ns	2.53ns	0.183	1.16 KB
#5476	`SendReceive`	net6.0	1.29μs	0.751ns	2.81ns	0.0161	1.14 KB
#5476	`SendReceive`	netcoreapp3.1	1.73μs	0.818ns	3.17ns	0.0156	1.14 KB
#5476	`SendReceive`	net472	2.11μs	1.94ns	7.51ns	0.183	1.16 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.6μs	1.51ns	5.64ns	0.0223	1.6 KB
master	`EnrichedLog`	netcoreapp3.1	3.84μs	0.73ns	2.73ns	0.0211	1.65 KB
master	`EnrichedLog`	net472	4.29μs	4.48ns	16.2ns	0.322	2.04 KB
#5476	`EnrichedLog`	net6.0	2.61μs	0.919ns	3.44ns	0.0234	1.6 KB
#5476	`EnrichedLog`	netcoreapp3.1	3.79μs	1.55ns	5.81ns	0.0208	1.65 KB
#5476	`EnrichedLog`	net472	4.3μs	1.16ns	4.48ns	0.322	2.04 KB

Benchmarks.Trace.SpanBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #5476

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net472	1.127	587.93	662.54

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	456ns	0.0971ns	0.363ns	0.00813	576 B
master	`StartFinishSpan`	netcoreapp3.1	554ns	0.193ns	0.748ns	0.00774	576 B
master	`StartFinishSpan`	net472	587ns	0.722ns	2.8ns	0.0917	578 B
master	`StartFinishScope`	net6.0	461ns	0.17ns	0.66ns	0.0098	696 B
master	`StartFinishScope`	netcoreapp3.1	676ns	0.367ns	1.42ns	0.00964	696 B
master	`StartFinishScope`	net472	847ns	0.382ns	1.43ns	0.104	658 B
#5476	`StartFinishSpan`	net6.0	426ns	0.15ns	0.582ns	0.008	576 B
#5476	`StartFinishSpan`	netcoreapp3.1	584ns	0.238ns	0.922ns	0.00792	576 B
#5476	`StartFinishSpan`	net472	662ns	0.507ns	1.96ns	0.0918	578 B
#5476	`StartFinishScope`	net6.0	468ns	0.0669ns	0.241ns	0.00966	696 B
#5476	`StartFinishScope`	netcoreapp3.1	666ns	0.291ns	1.09ns	0.00939	696 B
#5476	`StartFinishScope`	net472	863ns	0.694ns	2.69ns	0.105	658 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	668ns	0.559ns	2.17ns	0.00969	696 B
master	`RunOnMethodBegin`	netcoreapp3.1	919ns	0.313ns	1.21ns	0.00958	696 B
master	`RunOnMethodBegin`	net472	1.07μs	0.353ns	1.37ns	0.104	658 B
#5476	`RunOnMethodBegin`	net6.0	730ns	0.621ns	2.4ns	0.00988	696 B
#5476	`RunOnMethodBegin`	netcoreapp3.1	875ns	0.413ns	1.6ns	0.00926	696 B
#5476	`RunOnMethodBegin`	net472	1.13μs	0.491ns	1.9ns	0.104	658 B

andrewlock · 2024-04-22T12:26:00Z

Throughput/Crank Report:zap:

Throughput results for AspNetCoreSimpleController comparing the following branches/commits:

Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.

Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!

gantt
    title Throughput Linux x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5476) (11.939M)   : 0, 11938686
    master (11.881M)   : 0, 11880597
    benchmarks/2.9.0 (11.915M)   : 0, 11914657

    section Automatic
    This PR (5476) (8.063M)   : 0, 8062731
    master (8.047M)   : 0, 8046727
    benchmarks/2.9.0 (8.513M)   : 0, 8513066

    section Trace stats
    master (8.464M)   : 0, 8463773

    section Manual
    This PR (5476) (10.252M)   : 0, 10252228
    master (10.247M)   : 0, 10246957

    section Manual + Automatic
    This PR (5476) (7.512M)   : 0, 7511836
    master (7.578M)   : 0, 7577515

    section Version Conflict
    master (6.707M)   : 0, 6707065

gantt
    title Throughput Linux arm64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5476) (9.546M)   : 0, 9546024
    benchmarks/2.9.0 (9.711M)   : 0, 9711350

    section Automatic
    This PR (5476) (6.550M)   : 0, 6550413

    section Manual
    This PR (5476) (8.246M)   : 0, 8245631

    section Manual + Automatic
    This PR (5476) (6.217M)   : 0, 6217291

gantt
    title Throughput Windows x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5476) (10.032M)   : 0, 10031840
    master (10.099M)   : 0, 10099485

    section Automatic
    This PR (5476) (7.157M)   : 0, 7156646
    master (7.144M)   : 0, 7143660

    section Trace stats
    master (7.418M)   : 0, 7418021

    section Manual
    This PR (5476) (8.794M)   : 0, 8794122
    master (8.900M)   : 0, 8899902

    section Manual + Automatic
    This PR (5476) (6.921M)   : 0, 6920921
    master (6.862M)   : 0, 6861859

    section Version Conflict
    master (6.305M)   : 0, 6305249

profiler/src/ProfilerEngine/Datadog.Profiler.Native/OpSysTools.h

chrisnas · 2024-04-26T12:06:56Z

profiler/src/ProfilerEngine/Datadog.Profiler.Native.Linux/LinuxStackFramesCollector.cpp

@@ -109,6 +109,31 @@ StackSnapshotResultBuffer* LinuxStackFramesCollector::CollectStackSampleImplemen
            return GetStackSnapshotResult();
        }

+        // If there a timer associated to the managed thread, we have to disarm it.


Should be moved to line 89 so that self collect cases (exceptions, contention, ...) do not account stack walking to the application

profiler/test/Datadog.Profiler.Native.Tests/ConfigurationTest.cpp

chrisnas · 2024-04-26T12:15:54Z

profiler/src/ProfilerEngine/Datadog.Profiler.Native/StackSamplerLoop.cpp

@@ -76,6 +76,8 @@ StackSamplerLoop::StackSamplerLoop(
    Log::Info("Wall time sampled threads = ", _walltimeThreadsThreshold);
    Log::Info("Max CodeHotspots sampled threads = ", _codeHotspotsThreadsThreshold);
    Log::Info("Max CPU sampled threads = ", _cpuThreadsThreshold);
+    Log::Info("Cpu profiler is ", (_isCpuEnabled) ? "enabled" : "disabled");


Suggested change

Log::Info("Cpu profiler is ", (_isCpuEnabled) ? "enabled" : "disabled");

Log::Info("Manual Cpu profiler is ", (_isCpuEnabled) ? "enabled" : "disabled");

chrisnas · 2024-04-26T12:25:31Z

profiler/src/ProfilerEngine/Datadog.Profiler.Native.Linux/ProfilerSignalManager.cpp

-    return &signalManager;
+    if (signal < 1 || signal > 31)
+    {
+        throw std::invalid_argument(std::string("Signal argument is invalid ") + "(" + std::to_string(signal) + "). Value must be: 1 <= signal <= 31");


Avoid crashing the app: should assert to more easily find our mistake while debugging + log + return nullptr to be checked (in LinuxStackFramesCollector constructor and CorProfilerCallback init)

profiler/src/ProfilerEngine/Datadog.Profiler.Native.Linux/ProfilerSignalManager.cpp

profiler/src/ProfilerEngine/Datadog.Profiler.Native.Linux/TimerCreateCpuProfiler.cpp

chrisnas · 2024-04-26T13:39:03Z

profiler/src/ProfilerEngine/Datadog.Profiler.Native.Linux/TimerCreateCpuProfiler.cpp

+        return true;
+    }
+
+    // If the signal is higjacked, what to do?


Suggested change

// If the signal is higjacked, what to do?

// If the signal is hijacked, what to do?

profiler/src/ProfilerEngine/Datadog.Profiler.Native.Linux/TimerCreateCpuProfiler.cpp

datadog-ddstaging · 2024-06-17T14:22:44Z

Datadog Report

Branch report: gleocadie/add-timer-create-based-cpu-profiling
Commit report: 66f6868
Test service: dd-trace-dotnet

✅ 0 Failed, 344440 Passed, 2033 Skipped, 16h 7m 33.32s Total Time
❄️ 1 New Flaky

New Flaky Tests (1)

NoExceptions - Datadog.Trace.ClrProfiler.IntegrationTests.SmokeTests.RuntimeMetricsShutdownSmokeTest - Last Failure

Expand for error

 Expected no errors in smoke test: open(/proc/2390/mem) FAILED 2 (No such file or directory)
 
 Expected: True
 Actual:   False

chrisnas

LGTM

profiler/build/CpuWallTime.linux.json

profiler/build/crank/Samples.AspNetCoreSimpleController.yml

profiler/src/ProfilerEngine/Datadog.Profiler.Native.Linux/ProfilerSignalManager.cpp

chrisnas · 2024-06-25T12:36:18Z

profiler/src/ProfilerEngine/Datadog.Profiler.Native.Linux/TimerCreateCpuProfiler.h

+    IManagedThreadList* _pManagedThreadsList;
+    CpuTimeProvider* _pProvider;
+    CallstackProvider _callstackProvider;
+    std::atomic<ServiceState> _serviceState;


Should be removed because it is not more used (done in ServiceBase now)

profiler/src/ProfilerEngine/Datadog.Profiler.Native.Linux/TimerCreateCpuProfiler.cpp

chrisnas · 2024-06-25T13:02:57Z

profiler/src/ProfilerEngine/Datadog.Profiler.Native/CorProfilerCallback.cpp

-                    nullptr
-                }
-            };
+                {WStr("Microsoft-Windows-DotNETRuntime"),


add a line between { and the first element

github-actions bot added the area:profiler Issues related to the continous-profiler label Apr 19, 2024

gleocadie force-pushed the gleocadie/add-timer-create-based-cpu-profiling branch from 93658f6 to 88c285e Compare April 19, 2024 19:01

gleocadie force-pushed the gleocadie/add-timer-create-based-cpu-profiling branch from de46062 to b2a4323 Compare April 19, 2024 20:44

gleocadie force-pushed the gleocadie/add-timer-create-based-cpu-profiling branch from b3db22f to 46805c1 Compare April 23, 2024 11:45

gleocadie marked this pull request as ready for review April 23, 2024 11:51

gleocadie requested review from a team as code owners April 23, 2024 11:51

chrisnas reviewed Apr 26, 2024

View reviewed changes

gleocadie force-pushed the gleocadie/add-timer-create-based-cpu-profiling branch from 46805c1 to 588ddc2 Compare June 17, 2024 13:07

gleocadie force-pushed the gleocadie/add-timer-create-based-cpu-profiling branch from a57f047 to b7c3066 Compare June 21, 2024 09:02

gleocadie requested a review from chrisnas June 21, 2024 10:26

gleocadie force-pushed the gleocadie/add-timer-create-based-cpu-profiling branch 2 times, most recently from 5f98128 to 58c7c7e Compare June 21, 2024 18:32

chrisnas approved these changes Jun 25, 2024

View reviewed changes

gleocadie force-pushed the gleocadie/add-timer-create-based-cpu-profiling branch 2 times, most recently from db712f8 to 96174c9 Compare July 3, 2024 11:49

gleocadie added 9 commits July 5, 2024 10:53

Add knob to choose CPU profiler type

d2e8601

Add know to change CPU profiling interval

51a27fb

Add static thread_local CurrentTheadInfo

bdf56c5

Change Profiler Signal manager to handle multiple signals

6dcf79a

Improve tracing context retrieval

592247b

Add mmap-based memory resource

198197f

Add TimerCreate-based cpu profiler

b51fc44

Fix diverse stuff + add test

22a3896

Add Benchmark test

b4bfaea

gleocadie added 11 commits July 5, 2024 10:53

Add throughput test

5e095a9

clean up test

f3f4279

fix CI build

6902d61

Fix test and set default interval to 9ms

ea8276b

Oops

6047e5d

Clean up + add signal handler deregistration

a578cc4

WIP

419284b

Fix delayed startup timer registration/deregistration

66b1309

Address comment

f56b03a

fix unit test

d4b30bd

Address comment

66f6868

gleocadie force-pushed the gleocadie/add-timer-create-based-cpu-profiling branch from 96174c9 to 66f6868 Compare July 5, 2024 08:53

gleocadie merged commit b5fd279 into master Jul 5, 2024
65 checks passed

gleocadie deleted the gleocadie/add-timer-create-based-cpu-profiling branch July 5, 2024 13:40

github-actions bot added this to the vNext-v2 milestone Jul 5, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Profiler] Add `timer_create`-based CPU profiling on Linux #5476

[Profiler] Add `timer_create`-based CPU profiling on Linux #5476

gleocadie commented Apr 19, 2024 •

edited

Loading

datadog-ddstaging bot commented Apr 19, 2024 •

edited

Loading

andrewlock commented Apr 19, 2024 •

edited

Loading

andrewlock commented Apr 19, 2024 •

edited

Loading

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #5476

Raw results

Raw results

andrewlock commented Apr 22, 2024 •

edited

Loading

chrisnas Apr 26, 2024

gleocadie Jun 17, 2024

chrisnas Apr 26, 2024

gleocadie Jun 17, 2024

chrisnas Apr 26, 2024

gleocadie Jun 17, 2024

chrisnas Apr 26, 2024

gleocadie Jun 17, 2024

datadog-ddstaging bot commented Jun 17, 2024 •

edited

Loading

chrisnas left a comment

chrisnas Jun 25, 2024

gleocadie Jun 26, 2024

chrisnas Jun 25, 2024

gleocadie Jun 26, 2024

	Log::Info("Cpu profiler is ", (_isCpuEnabled) ? "enabled" : "disabled");
	Log::Info("Manual Cpu profiler is ", (_isCpuEnabled) ? "enabled" : "disabled");

	// If the signal is higjacked, what to do?
	// If the signal is hijacked, what to do?

[Profiler] Add timer_create-based CPU profiling on Linux #5476

[Profiler] Add timer_create-based CPU profiling on Linux #5476

Conversation

gleocadie commented Apr 19, 2024 • edited Loading

Summary of changes

Reason for change

Implementation details

Test coverage

Other details

datadog-ddstaging bot commented Apr 19, 2024 • edited Loading

Datadog Report

⌛ Performance Regressions vs Default Branch (1)

andrewlock commented Apr 19, 2024 • edited Loading

Execution-Time Benchmarks Report ⏱️

andrewlock commented Apr 19, 2024 • edited Loading

Benchmarks Report for tracer 🐌

Benchmark details

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #5476

Raw results

Raw results

andrewlock commented Apr 22, 2024 • edited Loading

Throughput/Crank Report:zap:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

datadog-ddstaging bot commented Jun 17, 2024 • edited Loading

Datadog Report

New Flaky Tests (1)

chrisnas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

[Profiler] Add `timer_create`-based CPU profiling on Linux #5476

[Profiler] Add `timer_create`-based CPU profiling on Linux #5476

gleocadie commented Apr 19, 2024 •

edited

Loading

datadog-ddstaging bot commented Apr 19, 2024 •

edited

Loading

andrewlock commented Apr 19, 2024 •

edited

Loading

andrewlock commented Apr 19, 2024 •

edited

Loading

andrewlock commented Apr 22, 2024 •

edited

Loading

datadog-ddstaging bot commented Jun 17, 2024 •

edited

Loading