[Dynamic Instrumentation] Temporary disable system tests #5411

shurivich · 2024-04-07T12:35:43Z

Summary of changes

Disable system tests for Dynamic Instrumentation

Reason for change

There is currently a bug in the master branch that breaks all system tests. This PR temporarily disables them for Dynamic Instrumentation to prevent anyone from being blocked until the issue is fixed.

datadog-ddstaging · 2024-04-07T12:50:12Z

Datadog Report

Branch report: shurivich/disable_debugger_st
Commit report: 8e5a91a
Test service: dd-trace-dotnet

✅ 0 Failed, 332911 Passed, 1592 Skipped, 40m 21.88s Wall Time

andrewlock · 2024-04-07T12:57:51Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5411) - mean (73ms)  : 65, 82
     .   : milestone, 73,
    master - mean (74ms)  : 63, 85
     .   : milestone, 74,

    section CallTarget+Inlining+NGEN
    This PR (5411) - mean (995ms)  : 976, 1015
     .   : milestone, 995,
    master - mean (993ms)  : 969, 1016
     .   : milestone, 993,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5411) - mean (110ms)  : 107, 113
     .   : milestone, 110,
    master - mean (109ms)  : 107, 111
     .   : milestone, 109,

    section CallTarget+Inlining+NGEN
    This PR (5411) - mean (711ms)  : 686, 736
     .   : milestone, 711,
    master - mean (711ms)  : 691, 731
     .   : milestone, 711,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5411) - mean (94ms)  : 91, 97
     .   : milestone, 94,
    master - mean (94ms)  : 91, 97
     .   : milestone, 94,

    section CallTarget+Inlining+NGEN
    This PR (5411) - mean (670ms)  : 645, 695
     .   : milestone, 670,
    master - mean (668ms)  : 639, 697
     .   : milestone, 668,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5411) - mean (186ms)  : 179, 192
     .   : milestone, 186,
    master - mean (186ms)  : 181, 192
     .   : milestone, 186,

    section CallTarget+Inlining+NGEN
    This PR (5411) - mean (1,076ms)  : 1052, 1100
     .   : milestone, 1076,
    master - mean (1,066ms)  : 1046, 1087
     .   : milestone, 1066,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5411) - mean (271ms)  : 267, 274
     .   : milestone, 271,
    master - mean (271ms)  : 266, 275
     .   : milestone, 271,

    section CallTarget+Inlining+NGEN
    This PR (5411) - mean (870ms)  : 843, 898
     .   : milestone, 870,
    master - mean (865ms)  : 836, 893
     .   : milestone, 865,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5411) - mean (260ms)  : 253, 266
     .   : milestone, 260,
    master - mean (258ms)  : 251, 265
     .   : milestone, 258,

    section CallTarget+Inlining+NGEN
    This PR (5411) - mean (848ms)  : 823, 873
     .   : milestone, 848,
    master - mean (848ms)  : 829, 868
     .   : milestone, 848,

andrewlock · 2024-04-07T13:11:27Z

Benchmarks Report for tracer 🐌

Benchmarks for #5411 compared to master:

2 benchmarks are slower, with geometric mean 1.143
All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	8.65μs	47.1ns	271ns	0.0228	0.00911	0	7.51 KB
master	`StartStopWithChild`	netcoreapp3.1	10.5μs	59.6ns	417ns	0.0263	0.0105	0	7.61 KB
master	`StartStopWithChild`	net472	16.8μs	68ns	263ns	1.32	0.329	0.101	8.05 KB
#5411	`StartStopWithChild`	net6.0	8.55μs	47.3ns	303ns	0.0262	0.0131	0	7.5 KB
#5411	`StartStopWithChild`	netcoreapp3.1	10.9μs	60.4ns	367ns	0.0312	0.0156	0	7.6 KB
#5411	`StartStopWithChild`	net472	17.1μs	48.1ns	186ns	1.37	0.338	0.127	8.05 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	450μs	149ns	557ns	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	599μs	382ns	1.48μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	817μs	308ns	1.19μs	0.408	3.3 KB
#5411	`WriteAndFlushEnrichedTraces`	net6.0	470μs	430ns	1.61μs	0	2.7 KB
#5411	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	613μs	140ns	504ns	0	2.7 KB
#5411	`WriteAndFlushEnrichedTraces`	net472	815μs	302ns	1.17μs	0.406	3.3 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net6.0	169μs	250ns	967ns	0.168	18.33 KB
master	`SendRequest`	netcoreapp3.1	193μs	410ns	1.59μs	0.193	20.49 KB
master	`SendRequest`	net472	0.000925ns	0.000278ns	0.00108ns	0	0 b
#5411	`SendRequest`	net6.0	171μs	155ns	602ns	0.173	18.33 KB
#5411	`SendRequest`	netcoreapp3.1	193μs	328ns	1.27μs	0.193	20.49 KB
#5411	`SendRequest`	net472	0.000737ns	0.000284ns	0.00106ns	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	548μs	2.75μs	12μs	0.584	0	0	41.45 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	639μs	853ns	3.3μs	0.319	0	0	41.86 KB
master	`WriteAndFlushEnrichedTraces`	net472	862μs	4.04μs	15.7μs	8.39	2.52	0.419	53.24 KB
#5411	`WriteAndFlushEnrichedTraces`	net6.0	542μs	622ns	2.41μs	0.556	0	0	41.46 KB
#5411	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	662μs	1.16μs	4.48μs	0.334	0	0	41.73 KB
#5411	`WriteAndFlushEnrichedTraces`	net472	864μs	3.87μs	15μs	8.39	2.52	0.419	53.22 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteNonQuery`	net6.0	1.18μs	0.426ns	1.54ns	0.0112	784 B
master	`ExecuteNonQuery`	netcoreapp3.1	1.52μs	0.702ns	2.72ns	0.0106	784 B
master	`ExecuteNonQuery`	net472	1.9μs	0.777ns	3.01ns	0.118	746 B
#5411	`ExecuteNonQuery`	net6.0	1.19μs	0.61ns	2.36ns	0.0107	784 B
#5411	`ExecuteNonQuery`	netcoreapp3.1	1.52μs	0.952ns	3.69ns	0.0106	784 B
#5411	`ExecuteNonQuery`	net472	1.83μs	0.873ns	3.26ns	0.119	746 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`CallElasticsearch`	net6.0	1.38μs	0.611ns	2.37ns	0.0145	1.01 KB
master	`CallElasticsearch`	netcoreapp3.1	1.64μs	1.54ns	5.98ns	0.0131	1.01 KB
master	`CallElasticsearch`	net472	2.49μs	1.06ns	4.09ns	0.161	1.02 KB
master	`CallElasticsearchAsync`	net6.0	1.41μs	0.968ns	3.75ns	0.0141	984 B
master	`CallElasticsearchAsync`	netcoreapp3.1	1.7μs	0.931ns	3.48ns	0.0143	1.06 KB
master	`CallElasticsearchAsync`	net472	2.69μs	1.63ns	6.3ns	0.171	1.08 KB
#5411	`CallElasticsearch`	net6.0	1.39μs	0.929ns	3.48ns	0.0139	1.01 KB
#5411	`CallElasticsearch`	netcoreapp3.1	1.54μs	0.839ns	3.25ns	0.0131	1.01 KB
#5411	`CallElasticsearch`	net472	2.54μs	0.687ns	2.57ns	0.161	1.02 KB
#5411	`CallElasticsearchAsync`	net6.0	1.3μs	0.557ns	2.16ns	0.0137	984 B
#5411	`CallElasticsearchAsync`	netcoreapp3.1	1.74μs	2.82ns	10.9ns	0.0147	1.06 KB
#5411	`CallElasticsearchAsync`	net472	2.71μs	1.44ns	5.57ns	0.17	1.08 KB

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net6.0	1.37μs	0.819ns	3.17ns	0.013	928 B
master	`ExecuteAsync`	netcoreapp3.1	1.61μs	0.819ns	2.95ns	0.012	928 B
master	`ExecuteAsync`	net472	1.98μs	1.02ns	3.82ns	0.141	891 B
#5411	`ExecuteAsync`	net6.0	1.33μs	0.555ns	2.07ns	0.0127	928 B
#5411	`ExecuteAsync`	netcoreapp3.1	1.6μs	0.973ns	3.64ns	0.012	928 B
#5411	`ExecuteAsync`	net472	1.91μs	2ns	7.73ns	0.141	891 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	4.28μs	2.15ns	8.05ns	0.0301	2.16 KB
master	`SendAsync`	netcoreapp3.1	5.14μs	3.32ns	12ns	0.0359	2.7 KB
master	`SendAsync`	net472	7.68μs	4.75ns	18.4ns	0.483	3.05 KB
#5411	`SendAsync`	net6.0	4.13μs	1.8ns	6.72ns	0.0292	2.16 KB
#5411	`SendAsync`	netcoreapp3.1	5.01μs	1.07ns	3.85ns	0.0375	2.7 KB
#5411	`SendAsync`	net472	7.85μs	1.76ns	6.11ns	0.482	3.05 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	1.62μs	2.86ns	11.1ns	0.023	1.65 KB
master	`EnrichedLog`	netcoreapp3.1	2.28μs	1.14ns	4.4ns	0.0215	1.65 KB
master	`EnrichedLog`	net472	2.7μs	1.81ns	6.77ns	0.249	1.57 KB
#5411	`EnrichedLog`	net6.0	1.51μs	1.08ns	4.06ns	0.0235	1.65 KB
#5411	`EnrichedLog`	netcoreapp3.1	2.24μs	1.65ns	6.16ns	0.0223	1.65 KB
#5411	`EnrichedLog`	net472	2.56μs	3.23ns	12.1ns	0.249	1.57 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net6.0	112μs	97.1ns	376ns	0.0558	0	4.23 KB
master	`EnrichedLog`	netcoreapp3.1	117μs	164ns	612ns	0.0585	0	4.23 KB
master	`EnrichedLog`	net472	146μs	51.7ns	200ns	0.654	0.218	4.41 KB
#5411	`EnrichedLog`	net6.0	114μs	93.2ns	349ns	0.057	0	4.23 KB
#5411	`EnrichedLog`	netcoreapp3.1	117μs	168ns	650ns	0	0	4.23 KB
#5411	`EnrichedLog`	net472	146μs	128ns	497ns	0.657	0.219	4.41 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.99μs	1.94ns	7.5ns	0.0311	2.21 KB
master	`EnrichedLog`	netcoreapp3.1	4.38μs	2.65ns	10.3ns	0.0303	2.21 KB
master	`EnrichedLog`	net472	4.96μs	2.5ns	9.7ns	0.32	2.02 KB
#5411	`EnrichedLog`	net6.0	2.84μs	1.82ns	7.05ns	0.0312	2.21 KB
#5411	`EnrichedLog`	netcoreapp3.1	4.17μs	2.34ns	9.08ns	0.0291	2.21 KB
#5411	`EnrichedLog`	net472	4.83μs	2.06ns	7.72ns	0.321	2.02 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendReceive`	net6.0	1.5μs	0.644ns	2.32ns	0.0165	1.18 KB
master	`SendReceive`	netcoreapp3.1	1.83μs	0.999ns	3.74ns	0.0155	1.18 KB
master	`SendReceive`	net472	2.33μs	3.16ns	12.2ns	0.186	1.18 KB
#5411	`SendReceive`	net6.0	1.48μs	0.786ns	3.05ns	0.0163	1.18 KB
#5411	`SendReceive`	netcoreapp3.1	1.89μs	1.33ns	5.17ns	0.016	1.18 KB
#5411	`SendReceive`	net472	2.26μs	1.99ns	7.43ns	0.187	1.18 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.79μs	0.603ns	2.26ns	0.0209	1.55 KB
master	`EnrichedLog`	netcoreapp3.1	3.94μs	1.7ns	6.6ns	0.0216	1.6 KB
master	`EnrichedLog`	net472	4.46μs	3.3ns	12.8ns	0.315	1.99 KB
#5411	`EnrichedLog`	net6.0	2.72μs	0.661ns	2.47ns	0.0219	1.55 KB
#5411	`EnrichedLog`	netcoreapp3.1	3.84μs	6.38ns	24.7ns	0.0211	1.6 KB
#5411	`EnrichedLog`	net472	4.61μs	1.64ns	6.12ns	0.316	1.99 KB

Benchmarks.Trace.SpanBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #5411

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net472	1.153	907.90	1,046.92

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	519ns	0.878ns	3.4ns	0.00763	552 B
master	`StartFinishSpan`	netcoreapp3.1	749ns	1.05ns	4.05ns	0.00717	552 B
master	`StartFinishSpan`	net472	812ns	1.38ns	5.36ns	0.088	554 B
master	`StartFinishScope`	net6.0	583ns	0.925ns	3.58ns	0.00936	672 B
master	`StartFinishScope`	netcoreapp3.1	898ns	1.96ns	7.58ns	0.00904	672 B
master	`StartFinishScope`	net472	909ns	1.83ns	7.07ns	0.101	634 B
#5411	`StartFinishSpan`	net6.0	473ns	0.702ns	2.63ns	0.00772	552 B
#5411	`StartFinishSpan`	netcoreapp3.1	772ns	1.43ns	5.54ns	0.00731	552 B
#5411	`StartFinishSpan`	net472	817ns	1.94ns	7.5ns	0.0879	554 B
#5411	`StartFinishScope`	net6.0	580ns	1.34ns	5.19ns	0.0094	672 B
#5411	`StartFinishScope`	netcoreapp3.1	809ns	0.993ns	3.72ns	0.00915	672 B
#5411	`StartFinishScope`	net472	1.05μs	2.03ns	7.87ns	0.101	634 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #5411

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin‑net6.0	1.133	630.91	714.97

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	630ns	1.45ns	5.62ns	0.00951	672 B
master	`RunOnMethodBegin`	netcoreapp3.1	958ns	2ns	7.76ns	0.00901	672 B
master	`RunOnMethodBegin`	net472	1.11μs	1.98ns	7.66ns	0.1	634 B
#5411	`RunOnMethodBegin`	net6.0	716ns	1.41ns	5.47ns	0.00952	672 B
#5411	`RunOnMethodBegin`	netcoreapp3.1	931ns	2.06ns	7.99ns	0.00919	672 B
#5411	`RunOnMethodBegin`	net472	1.09μs	1.54ns	5.98ns	0.101	634 B

OmerRaviv

LGTM

disable debugger scenarios

8e5a91a

shurivich requested a review from a team as a code owner April 7, 2024 12:35

github-actions bot added area:builds project files, build scripts, pipelines, versioning, releases, packages area:debugger labels Apr 7, 2024

shurivich self-assigned this Apr 7, 2024

OmerRaviv approved these changes Apr 7, 2024

View reviewed changes

shurivich merged commit 7780f07 into master Apr 8, 2024
54 of 57 checks passed

shurivich deleted the shurivich/disable_debugger_st branch April 8, 2024 07:03

github-actions bot added this to the vNext-v2 milestone Apr 8, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Dynamic Instrumentation] Temporary disable system tests #5411

[Dynamic Instrumentation] Temporary disable system tests #5411

shurivich commented Apr 7, 2024

datadog-ddstaging bot commented Apr 7, 2024 •

edited

Loading

andrewlock commented Apr 7, 2024

andrewlock commented Apr 7, 2024

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #5411

Raw results

Slower ⚠️ in #5411

Raw results

OmerRaviv left a comment

[Dynamic Instrumentation] Temporary disable system tests #5411

[Dynamic Instrumentation] Temporary disable system tests #5411

Conversation

shurivich commented Apr 7, 2024

Summary of changes

Reason for change

datadog-ddstaging bot commented Apr 7, 2024 • edited Loading

Datadog Report

andrewlock commented Apr 7, 2024

Execution-Time Benchmarks Report ⏱️

andrewlock commented Apr 7, 2024

Benchmarks Report for tracer 🐌

Benchmark details

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #5411

Raw results

Slower ⚠️ in #5411

Raw results

OmerRaviv left a comment

Choose a reason for hiding this comment

datadog-ddstaging bot commented Apr 7, 2024 •

edited

Loading