Tiny fix for crashing if `DD_PROFILER_ENABLED` is not set #4849

andrewlock · 2023-11-10T11:53:07Z

Summary of changes

Fix crash when trying to run Benchmarks.Trace locally

Reason for change

Was trying to run benchmarks locally, and getting crashes

Implementation details

Add a fallback default for enableProfiler so that we don't get NullReferenceExceptions

Test coverage

Ran it locally, it works

Other details

andrewlock · 2023-11-10T11:54:19Z

tracer/src/Datadog.Trace.BenchmarkDotNet/DatadogExtensions.cs

@@ -33,7 +33,7 @@ internal static IConfig WithDatadog(this IConfig config, bool? enableProfiler)
    {
        var cfg = config.AddLogger(DatadogSessionLogger.Default);

-        enableProfiler ??= (EnvironmentHelpers.GetEnvironmentVariable(ConfigurationKeys.ProfilingEnabled) ?? string.Empty).ToBoolean();
+        enableProfiler ??= (EnvironmentHelpers.GetEnvironmentVariable(ConfigurationKeys.ProfilingEnabled) ?? string.Empty).ToBoolean() ?? false;


If enableProfiler is null, and the ConfigurationKeys.ProfilingEnabled key isn't set, ToBoolean() returns a nullable, which we then try to unwrap on Line 47 and 💥

Then we probably don't need the ! at line 47 anymore?

datadog-ddstaging · 2023-11-10T12:19:01Z

Datadog Report

Branch report: andrew/benchmarkdotnet-fix
Commit report: 3833b49

✅ dd-trace-dotnet: 0 Failed, 0 New Flaky, 294306 Passed, 974 Skipped, 46m 44.94s Wall Time

andrewlock · 2023-11-10T12:21:48Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4849) - mean (72ms)  : 63, 81
     .   : milestone, 72,
    master - mean (70ms)  : 62, 79
     .   : milestone, 70,

    section CallTarget+Inlining+NGEN
    This PR (4849) - mean (1,003ms)  : 980, 1025
     .   : milestone, 1003,
    master - mean (993ms)  : 970, 1015
     .   : milestone, 993,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4849) - mean (106ms)  : 104, 109
     .   : milestone, 106,
    master - mean (106ms)  : 102, 110
     .   : milestone, 106,

    section CallTarget+Inlining+NGEN
    This PR (4849) - mean (689ms)  : 671, 707
     .   : milestone, 689,
    master - mean (689ms)  : 673, 706
     .   : milestone, 689,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4849) - mean (90ms)  : 87, 93
     .   : milestone, 90,
    master - mean (89ms)  : 85, 93
     .   : milestone, 89,

    section CallTarget+Inlining+NGEN
    This PR (4849) - mean (660ms)  : 635, 686
     .   : milestone, 660,
    master - mean (654ms)  : 628, 681
     .   : milestone, 654,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4849) - mean (187ms)  : 185, 190
     .   : milestone, 187,
    master - mean (187ms)  : 184, 190
     .   : milestone, 187,

    section CallTarget+Inlining+NGEN
    This PR (4849) - mean (1,097ms)  : 1078, 1116
     .   : milestone, 1097,
    master - mean (1,097ms)  : 1078, 1116
     .   : milestone, 1097,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4849) - mean (272ms)  : 267, 276
     .   : milestone, 272,
    master - mean (272ms)  : 268, 277
     .   : milestone, 272,

    section CallTarget+Inlining+NGEN
    This PR (4849) - mean (1,051ms)  : 1035, 1067
     .   : milestone, 1051,
    master - mean (1,054ms)  : 1027, 1081
     .   : milestone, 1054,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4849) - mean (263ms)  : 259, 266
     .   : milestone, 263,
    master - mean (262ms)  : 259, 265
     .   : milestone, 262,

    section CallTarget+Inlining+NGEN
    This PR (4849) - mean (1,031ms)  : 1009, 1053
     .   : milestone, 1031,
    master - mean (1,029ms)  : 1002, 1057
     .   : milestone, 1029,

kevingosse · 2023-11-10T12:55:00Z

tracer/src/Datadog.Trace.BenchmarkDotNet/DatadogExtensions.cs

@@ -33,7 +33,7 @@ internal static IConfig WithDatadog(this IConfig config, bool? enableProfiler)
    {
        var cfg = config.AddLogger(DatadogSessionLogger.Default);

-        enableProfiler ??= (EnvironmentHelpers.GetEnvironmentVariable(ConfigurationKeys.ProfilingEnabled) ?? string.Empty).ToBoolean();
+        enableProfiler ??= (EnvironmentHelpers.GetEnvironmentVariable(ConfigurationKeys.ProfilingEnabled) ?? string.Empty).ToBoolean() ?? false;


Then we probably don't need the ! at line 47 anymore?

andrewlock · 2023-11-10T17:13:46Z

Benchmarks Report 🐌

Benchmarks for #4849 compared to master:

1 benchmarks are faster, with geometric mean 1.183
3 benchmarks are slower, with geometric mean 1.133
All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	8.2μs	42.1ns	215ns	0.0242	0.00806	0	7.29 KB
master	`StartStopWithChild`	netcoreapp3.1	10.1μs	54.9ns	296ns	0.0254	0.0101	0	7.38 KB
master	`StartStopWithChild`	net472	15.6μs	40.5ns	157ns	1.32	0.35	0.122	7.67 KB
#4849	`StartStopWithChild`	net6.0	7.94μs	44.1ns	279ns	0.0241	0.0121	0	7.29 KB
#4849	`StartStopWithChild`	netcoreapp3.1	9.77μs	46.3ns	185ns	0.0235	0.0094	0	7.39 KB
#4849	`StartStopWithChild`	net472	15.8μs	50.4ns	195ns	1.3	0.349	0.119	7.65 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	470μs	152ns	568ns	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	638μs	918ns	3.44μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	810μs	291ns	1.13μs	0.403	3.3 KB
#4849	`WriteAndFlushEnrichedTraces`	net6.0	474μs	245ns	948ns	0	2.7 KB
#4849	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	678μs	123ns	444ns	0	2.7 KB
#4849	`WriteAndFlushEnrichedTraces`	net472	819μs	367ns	1.37μs	0.403	3.3 KB

Benchmarks.Trace.Asm.AppSecBodyBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`AllCycleSimpleBody`	net6.0	43.8μs	65.3ns	244ns	0.0216	0	2.02 KB
master	`AllCycleSimpleBody`	netcoreapp3.1	45.3μs	81.8ns	306ns	0.0223	0	2 KB
master	`AllCycleSimpleBody`	net472	47.3μs	19.9ns	74.5ns	0.308	0	2.07 KB
master	`AllCycleMoreComplexBody`	net6.0	205μs	112ns	418ns	0.103	0	8.37 KB
master	`AllCycleMoreComplexBody`	netcoreapp3.1	208μs	67.1ns	242ns	0.104	0	8.26 KB
master	`AllCycleMoreComplexBody`	net472	212μs	44.6ns	173ns	1.26	0	8.43 KB
master	`ObjectExtractorSimpleBody`	net6.0	128ns	0.0917ns	0.343ns	0.00396	0	280 B
master	`ObjectExtractorSimpleBody`	netcoreapp3.1	179ns	0.119ns	0.43ns	0.00368	0	272 B
master	`ObjectExtractorSimpleBody`	net472	156ns	0.0687ns	0.257ns	0.0446	0	281 B
master	`ObjectExtractorMoreComplexBody`	net6.0	2.71μs	1.08ns	4.06ns	0.0532	0	3.78 KB
master	`ObjectExtractorMoreComplexBody`	netcoreapp3.1	3.76μs	1.14ns	4.27ns	0.0489	0	3.69 KB
master	`ObjectExtractorMoreComplexBody`	net472	3.46μs	1.69ns	6.56ns	0.602	0.00518	3.8 KB
#4849	`AllCycleSimpleBody`	net6.0	44.1μs	27.6ns	107ns	0	0	2.02 KB
#4849	`AllCycleSimpleBody`	netcoreapp3.1	45.8μs	38.5ns	144ns	0.023	0	2 KB
#4849	`AllCycleSimpleBody`	net472	47μs	155ns	599ns	0.325	0	2.07 KB
#4849	`AllCycleMoreComplexBody`	net6.0	206μs	564ns	1.95μs	0.103	0	8.37 KB
#4849	`AllCycleMoreComplexBody`	netcoreapp3.1	209μs	134ns	517ns	0.104	0	8.26 KB
#4849	`AllCycleMoreComplexBody`	net472	214μs	89.7ns	324ns	1.28	0	8.43 KB
#4849	`ObjectExtractorSimpleBody`	net6.0	135ns	0.0771ns	0.288ns	0.00393	0	280 B
#4849	`ObjectExtractorSimpleBody`	netcoreapp3.1	189ns	0.127ns	0.476ns	0.00381	0	272 B
#4849	`ObjectExtractorSimpleBody`	net472	158ns	0.153ns	0.591ns	0.0446	0	281 B
#4849	`ObjectExtractorMoreComplexBody`	net6.0	2.77μs	0.769ns	2.88ns	0.053	0	3.78 KB
#4849	`ObjectExtractorMoreComplexBody`	netcoreapp3.1	3.76μs	0.913ns	3.42ns	0.0507	0	3.69 KB
#4849	`ObjectExtractorMoreComplexBody`	net472	3.58μs	2.66ns	9.96ns	0.602	0.00708	3.8 KB

Benchmarks.Trace.Asm.AppSecWafBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`RunWaf(args=NestedMap (10))`	net6.0	12.9μs	9.06ns	35.1ns	0.129	0	9.42 KB
master	`RunWaf(args=NestedMap (10))`	netcoreapp3.1	20.3μs	3.64ns	12.6ns	0.122	0	9.41 KB
master	`RunWaf(args=NestedMap (10))`	net472	29.8μs	9.8ns	38ns	1.49	0	9.47 KB
master	`RunWafWithAttack(args=Neste(...)tack) [22])`	net6.0	77.7μs	27.8ns	108ns	0.212	0	15.76 KB
master	`RunWafWithAttack(args=Neste(...)tack) [22])`	netcoreapp3.1	86.8μs	111ns	432ns	0.216	0	15.71 KB
master	`RunWafWithAttack(args=Neste(...)tack) [22])`	net472	99.6μs	269ns	1.04μs	2.51	0	16.03 KB
master	`RunWaf(args=NestedMap (100))`	net6.0	24.6μs	8.72ns	31.4ns	0.27	0	19.66 KB
master	`RunWaf(args=NestedMap (100))`	netcoreapp3.1	39μs	11.6ns	44.9ns	0.273	0	20.41 KB
master	`RunWaf(args=NestedMap (100))`	net472	57.5μs	35.7ns	138ns	3.27	0.0574	20.62 KB
master	`RunWafWithAttack(args=Neste(...)tack) [23])`	net6.0	94.8μs	171ns	615ns	0.33	0	26 KB
master	`RunWafWithAttack(args=Neste(...)tack) [23])`	netcoreapp3.1	110μs	223ns	864ns	0.327	0	26.71 KB
master	`RunWafWithAttack(args=Neste(...)tack) [23])`	net472	131μs	228ns	883ns	4.27	0.0648	27.19 KB
master	`RunWaf(args=NestedMap (20))`	net6.0	24.5μs	33ns	128ns	0.266	0	19.39 KB
master	`RunWaf(args=NestedMap (20))`	netcoreapp3.1	38.6μs	10.3ns	38.5ns	0.27	0	19.83 KB
master	`RunWaf(args=NestedMap (20))`	net472	56.5μs	51.8ns	200ns	3.17	0.0561	20.03 KB
master	`RunWafWithAttack(args=Neste(...)tack) [22])`	net6.0	91.2μs	20.3ns	70.2ns	0.365	0	25.74 KB
master	`RunWafWithAttack(args=Neste(...)tack) [22])`	netcoreapp3.1	109μs	260ns	1.01μs	0.323	0	26.14 KB
master	`RunWafWithAttack(args=Neste(...)tack) [22])`	net472	132μs	56.2ns	218ns	4.21	0.0658	26.59 KB
#4849	`RunWaf(args=NestedMap (10))`	net6.0	13μs	6.18ns	23.9ns	0.131	0	9.42 KB
#4849	`RunWaf(args=NestedMap (10))`	netcoreapp3.1	20μs	6.33ns	23.7ns	0.12	0	9.41 KB
#4849	`RunWaf(args=NestedMap (10))`	net472	30.2μs	7.84ns	29.3ns	1.5	0.0151	9.47 KB
#4849	`RunWafWithAttack(args=Neste(...)tack) [22])`	net6.0	78μs	112ns	418ns	0.193	0	15.76 KB
#4849	`RunWafWithAttack(args=Neste(...)tack) [22])`	netcoreapp3.1	86.4μs	106ns	397ns	0.214	0	15.71 KB
#4849	`RunWafWithAttack(args=Neste(...)tack) [22])`	net472	99.1μs	199ns	770ns	2.5	0	16.03 KB
#4849	`RunWaf(args=NestedMap (100))`	net6.0	24.2μs	21.9ns	84.7ns	0.279	0	19.66 KB
#4849	`RunWaf(args=NestedMap (100))`	netcoreapp3.1	39.2μs	9.51ns	35.6ns	0.274	0	20.41 KB
#4849	`RunWaf(args=NestedMap (100))`	net472	56.5μs	27.6ns	103ns	3.26	0.0567	20.62 KB
#4849	`RunWafWithAttack(args=Neste(...)tack) [23])`	net6.0	92.9μs	33.8ns	122ns	0.328	0	26 KB
#4849	`RunWafWithAttack(args=Neste(...)tack) [23])`	netcoreapp3.1	109μs	90.8ns	340ns	0.326	0	26.71 KB
#4849	`RunWafWithAttack(args=Neste(...)tack) [23])`	net472	134μs	445ns	1.72μs	4.29	0.0659	27.19 KB
#4849	`RunWaf(args=NestedMap (20))`	net6.0	24.3μs	22.7ns	81.9ns	0.268	0	19.39 KB
#4849	`RunWaf(args=NestedMap (20))`	netcoreapp3.1	39.2μs	10.8ns	40.3ns	0.254	0	19.83 KB
#4849	`RunWaf(args=NestedMap (20))`	net472	56.4μs	19.6ns	76ns	3.16	0.0564	20.03 KB
#4849	`RunWafWithAttack(args=Neste(...)tack) [22])`	net6.0	91.9μs	32.4ns	126ns	0.364	0	25.74 KB
#4849	`RunWafWithAttack(args=Neste(...)tack) [22])`	netcoreapp3.1	110μs	247ns	854ns	0.329	0	26.14 KB
#4849	`RunWafWithAttack(args=Neste(...)tack) [22])`	net472	130μs	236ns	915ns	4.19	0.0644	26.59 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net6.0	171μs	110ns	410ns	0.171	18.04 KB
master	`SendRequest`	netcoreapp3.1	190μs	267ns	1E+03ns	0.191	20.2 KB
master	`SendRequest`	net472	6.42E‑05ns	5.18E‑05ns	0.000194ns	0	0 b
#4849	`SendRequest`	net6.0	171μs	163ns	632ns	0.171	18.04 KB
#4849	`SendRequest`	netcoreapp3.1	193μs	167ns	626ns	0.193	20.2 KB
#4849	`SendRequest`	net472	0.000736ns	0.000281ns	0.00109ns	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	554μs	1.15μs	4.46μs	0.546	0	0	41.81 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	637μs	1.21μs	4.68μs	0.317	0	0	41.82 KB
master	`WriteAndFlushEnrichedTraces`	net472	831μs	3.48μs	13.5μs	8.08	2.55	0.425	53.26 KB
#4849	`WriteAndFlushEnrichedTraces`	net6.0	556μs	298ns	1.08μs	0.532	0	0	41.91 KB
#4849	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	667μs	1.51μs	5.85μs	0.329	0	0	41.77 KB
#4849	`WriteAndFlushEnrichedTraces`	net472	871μs	4.28μs	18.2μs	8.13	2.57	0.428	53.27 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteNonQuery`	net6.0	945ns	0.398ns	1.54ns	0.0105	768 B
master	`ExecuteNonQuery`	netcoreapp3.1	1.33μs	0.888ns	3.32ns	0.0104	768 B
master	`ExecuteNonQuery`	net472	1.6μs	4.62ns	17.9ns	0.116	730 B
#4849	`ExecuteNonQuery`	net6.0	967ns	0.466ns	1.74ns	0.0107	768 B
#4849	`ExecuteNonQuery`	netcoreapp3.1	1.28μs	0.537ns	1.94ns	0.0102	768 B
#4849	`ExecuteNonQuery`	net472	1.55μs	0.45ns	1.68ns	0.116	730 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`CallElasticsearch`	net6.0	1.21μs	0.479ns	1.79ns	0.0133	936 B
master	`CallElasticsearch`	netcoreapp3.1	1.33μs	1.46ns	5.07ns	0.0131	936 B
master	`CallElasticsearch`	net472	2.39μs	0.536ns	1.86ns	0.151	955 B
master	`CallElasticsearchAsync`	net6.0	1.21μs	0.375ns	1.4ns	0.0127	912 B
master	`CallElasticsearchAsync`	netcoreapp3.1	1.46μs	0.364ns	1.31ns	0.0134	984 B
master	`CallElasticsearchAsync`	net472	2.41μs	0.601ns	2.33ns	0.16	1.01 KB
#4849	`CallElasticsearch`	net6.0	1.16μs	0.729ns	2.82ns	0.0132	936 B
#4849	`CallElasticsearch`	netcoreapp3.1	1.36μs	0.803ns	3ns	0.0123	936 B
#4849	`CallElasticsearch`	net472	2.36μs	0.49ns	1.9ns	0.151	955 B
#4849	`CallElasticsearchAsync`	net6.0	1.28μs	0.487ns	1.82ns	0.0127	912 B
#4849	`CallElasticsearchAsync`	netcoreapp3.1	1.49μs	0.615ns	2.38ns	0.0128	984 B
#4849	`CallElasticsearchAsync`	net472	2.56μs	0.683ns	2.46ns	0.16	1.01 KB

Benchmarks.Trace.GraphQLBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #4849

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.GraphQLBenchmark.ExecuteAsync‑netcoreapp3.1	1.143	1,377.26	1,573.66

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net6.0	1.19μs	0.534ns	2ns	0.0126	912 B
master	`ExecuteAsync`	netcoreapp3.1	1.38μs	0.94ns	3.64ns	0.0124	912 B
master	`ExecuteAsync`	net472	1.61μs	0.901ns	3.49ns	0.139	875 B
#4849	`ExecuteAsync`	net6.0	1.31μs	0.646ns	2.42ns	0.0125	912 B
#4849	`ExecuteAsync`	netcoreapp3.1	1.57μs	0.86ns	3.33ns	0.0126	912 B
#4849	`ExecuteAsync`	net472	1.75μs	0.777ns	2.8ns	0.139	875 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	3.88μs	1.66ns	6.2ns	0.0253	1.9 KB
master	`SendAsync`	netcoreapp3.1	4.52μs	6.73ns	26.1ns	0.0314	2.43 KB
master	`SendAsync`	net472	6.96μs	2.7ns	10.5ns	0.475	2.99 KB
#4849	`SendAsync`	net6.0	3.81μs	1.24ns	4.8ns	0.0268	1.9 KB
#4849	`SendAsync`	netcoreapp3.1	4.38μs	1.65ns	5.95ns	0.0331	2.43 KB
#4849	`SendAsync`	net472	7.07μs	4.8ns	18.6ns	0.475	2.99 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	1.34μs	0.422ns	1.58ns	0.0222	1.57 KB
master	`EnrichedLog`	netcoreapp3.1	1.94μs	2.81ns	10.1ns	0.0213	1.57 KB
master	`EnrichedLog`	net472	2.24μs	1.29ns	4.99ns	0.236	1.49 KB
#4849	`EnrichedLog`	net6.0	1.42μs	0.717ns	2.78ns	0.022	1.57 KB
#4849	`EnrichedLog`	netcoreapp3.1	1.93μs	0.997ns	3.73ns	0.0209	1.57 KB
#4849	`EnrichedLog`	net472	2.36μs	0.833ns	3.23ns	0.237	1.49 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net6.0	113μs	80.9ns	303ns	0	0	4.21 KB
master	`EnrichedLog`	netcoreapp3.1	118μs	157ns	607ns	0	0	4.21 KB
master	`EnrichedLog`	net472	149μs	322ns	1.25μs	0.662	0.221	4.38 KB
#4849	`EnrichedLog`	net6.0	112μs	150ns	582ns	0.0558	0	4.21 KB
#4849	`EnrichedLog`	netcoreapp3.1	116μs	82.8ns	299ns	0.0579	0	4.21 KB
#4849	`EnrichedLog`	net472	149μs	94.2ns	340ns	0.669	0.223	4.38 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.74μs	1.24ns	4.64ns	0.029	2.13 KB
master	`EnrichedLog`	netcoreapp3.1	3.84μs	2.14ns	8.01ns	0.0288	2.13 KB
master	`EnrichedLog`	net472	4.59μs	1.37ns	5.11ns	0.306	1.93 KB
#4849	`EnrichedLog`	net6.0	2.99μs	1.17ns	4.54ns	0.0298	2.13 KB
#4849	`EnrichedLog`	netcoreapp3.1	3.72μs	1.12ns	4.04ns	0.0279	2.13 KB
#4849	`EnrichedLog`	net472	4.5μs	1.46ns	5.48ns	0.307	1.93 KB

Benchmarks.Trace.RedisBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #4849

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.RedisBenchmark.SendReceive‑net6.0	1.183	1,449.67	1,225.51

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendReceive`	net6.0	1.45μs	0.962ns	3.73ns	0.0152	1.1 KB
master	`SendReceive`	netcoreapp3.1	1.67μs	7.89ns	30.6ns	0.0143	1.1 KB
master	`SendReceive`	net472	1.93μs	2.85ns	11.1ns	0.177	1.12 KB
#4849	`SendReceive`	net6.0	1.23μs	0.462ns	1.73ns	0.0153	1.1 KB
#4849	`SendReceive`	netcoreapp3.1	1.59μs	2.35ns	8.79ns	0.0152	1.1 KB
#4849	`SendReceive`	net472	1.8μs	1.03ns	3.97ns	0.176	1.12 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.63μs	0.951ns	3.56ns	0.0209	1.53 KB
master	`EnrichedLog`	netcoreapp3.1	3.65μs	1.14ns	4.41ns	0.0201	1.58 KB
master	`EnrichedLog`	net472	4.13μs	1.96ns	7.61ns	0.31	1.96 KB
#4849	`EnrichedLog`	net6.0	2.63μs	1.31ns	5.07ns	0.0211	1.53 KB
#4849	`EnrichedLog`	netcoreapp3.1	3.51μs	0.662ns	2.48ns	0.0211	1.58 KB
#4849	`EnrichedLog`	net472	3.93μs	1.28ns	4.97ns	0.31	1.96 KB

Benchmarks.Trace.SpanBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #4849

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net472	1.133	609.30	690.56
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net6.0	1.124	516.74	580.84

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	411ns	0.323ns	1.25ns	0.00757	536 B
master	`StartFinishSpan`	netcoreapp3.1	496ns	0.141ns	0.529ns	0.00739	536 B
master	`StartFinishSpan`	net472	609ns	0.193ns	0.748ns	0.0853	538 B
master	`StartFinishScope`	net6.0	517ns	0.188ns	0.728ns	0.00935	656 B
master	`StartFinishScope`	netcoreapp3.1	710ns	0.248ns	0.929ns	0.00885	656 B
master	`StartFinishScope`	net472	860ns	0.512ns	1.98ns	0.0981	618 B
#4849	`StartFinishSpan`	net6.0	454ns	0.196ns	0.758ns	0.00754	536 B
#4849	`StartFinishSpan`	netcoreapp3.1	517ns	0.337ns	1.22ns	0.00739	536 B
#4849	`StartFinishSpan`	net472	691ns	0.221ns	0.856ns	0.0851	538 B
#4849	`StartFinishScope`	net6.0	581ns	0.224ns	0.868ns	0.00914	656 B
#4849	`StartFinishScope`	netcoreapp3.1	780ns	0.885ns	3.43ns	0.00885	656 B
#4849	`StartFinishScope`	net472	839ns	0.662ns	2.56ns	0.0979	618 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	615ns	0.306ns	1.18ns	0.00946	656 B
master	`RunOnMethodBegin`	netcoreapp3.1	834ns	0.227ns	0.88ns	0.00876	656 B
master	`RunOnMethodBegin`	net472	1μs	0.426ns	1.65ns	0.0979	618 B
#4849	`RunOnMethodBegin`	net6.0	622ns	0.273ns	0.945ns	0.00909	656 B
#4849	`RunOnMethodBegin`	netcoreapp3.1	779ns	0.279ns	1.04ns	0.00884	656 B
#4849	`RunOnMethodBegin`	net472	999ns	0.407ns	1.58ns	0.0981	618 B

andrewlock · 2023-11-11T01:51:46Z

Throughput/Crank Report:zap:

Throughput results for AspNetCoreSimpleController comparing the following branches/commits:

Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.

Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!

gantt
    title Throughput Linux x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4849) (11.075M)   : 0, 11074747
    master (11.365M)   : 0, 11364660
    benchmarks/2.9.0 (11.650M)   : 0, 11650156

    section Automatic
    This PR (4849) (7.665M)   : 0, 7665312
    master (7.952M)   : 0, 7952033
    benchmarks/2.9.0 (8.463M)   : 0, 8463497

    section Trace stats
    This PR (4849) (7.983M)   : 0, 7982719
    master (8.125M)   : 0, 8124896

    section Manual
    This PR (4849) (9.618M)   : 0, 9618377
    master (9.996M)   : 0, 9996189

    section Manual + Automatic
    This PR (4849) (7.059M)   : crit ,0, 7058592
    master (7.477M)   : 0, 7476818

    section Version Conflict
    This PR (4849) (6.625M)   : 0, 6625225
    master (6.908M)   : 0, 6907880

gantt
    title Throughput Linux arm64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4849) (9.544M)   : 0, 9544411
    master (9.555M)   : 0, 9554556
    benchmarks/2.9.0 (9.454M)   : 0, 9454103

    section Automatic
    This PR (4849) (6.611M)   : 0, 6610609
    master (6.532M)   : 0, 6531809

    section Trace stats
    This PR (4849) (7.059M)   : 0, 7059411
    master (6.948M)   : 0, 6948235

    section Manual
    This PR (4849) (8.480M)   : 0, 8480163
    master (8.325M)   : 0, 8325246

    section Manual + Automatic
    This PR (4849) (6.228M)   : 0, 6228348
    master (6.277M)   : 0, 6277324

    section Version Conflict
    This PR (4849) (5.742M)   : 0, 5741768
    master (5.924M)   : 0, 5924344

gantt
    title Throughput Windows x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4849) (8.942M)   : 0, 8941980
    master (10.211M)   : 0, 10211482
    benchmarks/2.9.0 (9.974M)   : 0, 9974322

    section Automatic
    This PR (4849) (7.226M)   : 0, 7225749
    master (7.507M)   : 0, 7507323
    benchmarks/2.9.0 (7.408M)   : 0, 7408367

    section Trace stats
    This PR (4849) (7.661M)   : 0, 7660979
    master (7.883M)   : 0, 7882755

    section Manual
    This PR (4849) (9.371M)   : 0, 9370552
    master (9.456M)   : 0, 9456127

    section Manual + Automatic
    This PR (4849) (6.903M)   : crit ,0, 6903364
    master (7.282M)   : 0, 7282409

    section Version Conflict
    This PR (4849) (6.403M)   : 0, 6402676
    master (6.615M)   : 0, 6615388

gantt
    title Throughput Linux x64 (ASM) (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    master (7.687M)   : 0, 7687290
    benchmarks/2.9.0 (7.940M)   : 0, 7940428

    section No attack
    master (2.179M)   : 0, 2179090
    benchmarks/2.9.0 (3.229M)   : 0, 3229263

    section Attack
    master (1.706M)   : 0, 1706183
    benchmarks/2.9.0 (2.541M)   : 0, 2540724

    section Blocking
    master (3.530M)   : 0, 3530105

    section IAST default
    master (6.727M)   : 0, 6726784

    section IAST full
    master (6.129M)   : 0, 6128654

    section Base vuln
    master (0.990M)   : 0, 990312

    section IAST vuln
    master (0.867M)   : 0, 867410

gleocadie

LGTM

Tiny fix for crashing if DD_PROFILER_ENABLED is not set

62eb66b

andrewlock added type:bug area:benchmarks Benchmarks, throughput tests, Crank, Bombardier, etc labels Nov 10, 2023

andrewlock requested a review from a team as a code owner November 10, 2023 11:53

andrewlock commented Nov 10, 2023

View reviewed changes

kevingosse approved these changes Nov 10, 2023

View reviewed changes

Update DatadogExtensions.cs

3833b49

zacharycmontoya approved these changes Nov 10, 2023

View reviewed changes

lucaspimentel approved these changes Nov 10, 2023

View reviewed changes

gleocadie approved these changes Nov 13, 2023

View reviewed changes

andrewlock merged commit d5bfe48 into master Nov 13, 2023
54 checks passed

andrewlock deleted the andrew/benchmarkdotnet-fix branch November 13, 2023 09:26

github-actions bot added this to the vNext milestone Nov 13, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tiny fix for crashing if `DD_PROFILER_ENABLED` is not set #4849

Tiny fix for crashing if `DD_PROFILER_ENABLED` is not set #4849

andrewlock commented Nov 10, 2023

andrewlock Nov 10, 2023

kevingosse Nov 10, 2023

datadog-ddstaging bot commented Nov 10, 2023 •

edited

Loading

andrewlock commented Nov 10, 2023 •

edited

Loading

kevingosse Nov 10, 2023

andrewlock commented Nov 10, 2023

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #4849

Raw results

Raw results

Raw results

Raw results

Raw results

Faster 🎉 in #4849

Raw results

Raw results

Slower ⚠️ in #4849

Raw results

Raw results

andrewlock commented Nov 11, 2023

gleocadie left a comment

Tiny fix for crashing if DD_PROFILER_ENABLED is not set #4849

Tiny fix for crashing if DD_PROFILER_ENABLED is not set #4849

Conversation

andrewlock commented Nov 10, 2023

Summary of changes

Reason for change

Implementation details

Test coverage

Other details

andrewlock Nov 10, 2023

Choose a reason for hiding this comment

kevingosse Nov 10, 2023

Choose a reason for hiding this comment

datadog-ddstaging bot commented Nov 10, 2023 • edited Loading

Datadog Report

andrewlock commented Nov 10, 2023 • edited Loading

Execution-Time Benchmarks Report ⏱️

kevingosse Nov 10, 2023

Choose a reason for hiding this comment

andrewlock commented Nov 10, 2023

Benchmarks Report 🐌

Benchmark details

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #4849

Raw results

Raw results

Raw results

Raw results

Raw results

Faster 🎉 in #4849

Raw results

Raw results

Slower ⚠️ in #4849

Raw results

Raw results

andrewlock commented Nov 11, 2023

Throughput/Crank Report:zap:

gleocadie left a comment

Choose a reason for hiding this comment

Tiny fix for crashing if `DD_PROFILER_ENABLED` is not set #4849

Tiny fix for crashing if `DD_PROFILER_ENABLED` is not set #4849

datadog-ddstaging bot commented Nov 10, 2023 •

edited

Loading

andrewlock commented Nov 10, 2023 •

edited

Loading