Remove direct reference to Samples.AspNetCore.RazorPages from integration test project #5402

andrewlock · 2024-04-04T15:24:21Z

Summary of changes

Removes the direct reference to Samples.AspNetCore.RazorPages from Datadog.Trace.IntegrationTests

Reason for change

It makes the build more complicated, it adds coupling between tests that we may not want, and it's overall not necessary.

Implementation details

Remove the reference, add explicit Razor Pages to the test project directly

Test coverage

Unchanged

Other details

I tried linking in files from the test app instead, but that required updating a bunch of the files (the sample apps aren't covered by the same analyzers, so requires moving a bunch of stuff around). It would also mean the separate tests are coupled too.

datadog-ddstaging · 2024-04-04T15:55:00Z

Datadog Report

Branch report: andrew/ci/remove-direct-razor-pages-reference
Commit report: fa42d01
Test service: dd-trace-dotnet

✅ 0 Failed, 334463 Passed, 1519 Skipped, 45m 44.44s Wall Time

lucaspimentel · 2024-04-04T16:14:50Z

Thank you! Just one small comment. Instead of Pages in tracer/test/Datadog.Trace.IntegrationTests/Pages/, can we expand that name? Maybe .../AspNetCoreRazorPages/, or at least .../RazorPages/. "Pages" alone it not very clear.

lucaspimentel

THANK YOU

andrewlock · 2024-04-04T16:27:07Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5402) - mean (74ms)  : 65, 83
     .   : milestone, 74,
    master - mean (74ms)  : 66, 82
     .   : milestone, 74,

    section CallTarget+Inlining+NGEN
    This PR (5402) - mean (995ms)  : 968, 1022
     .   : milestone, 995,
    master - mean (1,005ms)  : 978, 1032
     .   : milestone, 1005,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5402) - mean (110ms)  : 107, 113
     .   : milestone, 110,
    master - mean (109ms)  : 106, 112
     .   : milestone, 109,

    section CallTarget+Inlining+NGEN
    This PR (5402) - mean (717ms)  : 693, 741
     .   : milestone, 717,
    master - mean (720ms)  : 692, 747
     .   : milestone, 720,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5402) - mean (95ms)  : 91, 99
     .   : milestone, 95,
    master - mean (95ms)  : 92, 98
     .   : milestone, 95,

    section CallTarget+Inlining+NGEN
    This PR (5402) - mean (671ms)  : 647, 694
     .   : milestone, 671,
    master - mean (674ms)  : 651, 696
     .   : milestone, 674,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5402) - mean (187ms)  : 182, 192
     .   : milestone, 187,
    master - mean (189ms)  : 186, 192
     .   : milestone, 189,

    section CallTarget+Inlining+NGEN
    This PR (5402) - mean (1,074ms)  : 1049, 1099
     .   : milestone, 1074,
    master - mean (1,085ms)  : 1059, 1111
     .   : milestone, 1085,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5402) - mean (270ms)  : 264, 275
     .   : milestone, 270,
    master - mean (273ms)  : 268, 278
     .   : milestone, 273,

    section CallTarget+Inlining+NGEN
    This PR (5402) - mean (870ms)  : 839, 901
     .   : milestone, 870,
    master - mean (875ms)  : 850, 900
     .   : milestone, 875,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5402) - mean (260ms)  : 256, 264
     .   : milestone, 260,
    master - mean (262ms)  : 258, 266
     .   : milestone, 262,

    section CallTarget+Inlining+NGEN
    This PR (5402) - mean (854ms)  : 830, 879
     .   : milestone, 854,
    master - mean (859ms)  : 831, 888
     .   : milestone, 859,

andrewlock · 2024-04-04T17:53:28Z

Benchmarks Report for tracer 🐌

Benchmarks for #5402 compared to master:

1 benchmarks are slower, with geometric mean 1.156
All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	8.79μs	45.2ns	256ns	0.0222	0.0089	0	7.51 KB
master	`StartStopWithChild`	netcoreapp3.1	10.5μs	58.8ns	377ns	0.0266	0.0106	0	7.6 KB
master	`StartStopWithChild`	net472	16.8μs	67.6ns	262ns	1.35	0.36	0.109	8.06 KB
#5402	`StartStopWithChild`	net6.0	8.55μs	47.9ns	299ns	0.0256	0.0128	0	7.5 KB
#5402	`StartStopWithChild`	netcoreapp3.1	10.9μs	54.5ns	250ns	0.0264	0.0106	0	7.6 KB
#5402	`StartStopWithChild`	net472	17.1μs	53.7ns	208ns	1.34	0.326	0.109	8.06 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	478μs	1.05μs	4.06μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	604μs	299ns	1.03μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	818μs	442ns	1.65μs	0.408	3.3 KB
#5402	`WriteAndFlushEnrichedTraces`	net6.0	464μs	127ns	491ns	0	2.7 KB
#5402	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	606μs	164ns	613ns	0	2.7 KB
#5402	`WriteAndFlushEnrichedTraces`	net472	815μs	210ns	787ns	0.406	3.3 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net6.0	172μs	300ns	1.16μs	0.259	18.33 KB
master	`SendRequest`	netcoreapp3.1	194μs	402ns	1.56μs	0.193	20.49 KB
master	`SendRequest`	net472	0.000226ns	0.000118ns	0.00044ns	0	0 b
#5402	`SendRequest`	net6.0	174μs	267ns	1.03μs	0.174	18.33 KB
#5402	`SendRequest`	netcoreapp3.1	194μs	313ns	1.21μs	0.194	20.49 KB
#5402	`SendRequest`	net472	0.00144ns	0.000277ns	0.00107ns	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	542μs	1.7μs	6.57μs	0.539	0	0	41.68 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	649μs	725ns	2.71μs	0.329	0	0	41.76 KB
master	`WriteAndFlushEnrichedTraces`	net472	879μs	4.38μs	19.6μs	8.08	2.55	0.425	53.25 KB
#5402	`WriteAndFlushEnrichedTraces`	net6.0	550μs	1.63μs	6.32μs	0.566	0	0	41.81 KB
#5402	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	649μs	2.07μs	8μs	0.327	0	0	41.72 KB
#5402	`WriteAndFlushEnrichedTraces`	net472	851μs	3.67μs	14.2μs	8.33	2.5	0.417	53.25 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteNonQuery`	net6.0	1.21μs	0.329ns	1.27ns	0.011	784 B
master	`ExecuteNonQuery`	netcoreapp3.1	1.55μs	0.816ns	3.16ns	0.0103	784 B
master	`ExecuteNonQuery`	net472	1.83μs	1.64ns	6.37ns	0.118	746 B
#5402	`ExecuteNonQuery`	net6.0	1.16μs	0.376ns	1.46ns	0.011	784 B
#5402	`ExecuteNonQuery`	netcoreapp3.1	1.62μs	0.319ns	1.1ns	0.0105	784 B
#5402	`ExecuteNonQuery`	net472	1.78μs	2.58ns	9.99ns	0.118	746 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`CallElasticsearch`	net6.0	1.23μs	0.686ns	2.47ns	0.0142	1.01 KB
master	`CallElasticsearch`	netcoreapp3.1	1.61μs	0.634ns	2.37ns	0.0136	1.01 KB
master	`CallElasticsearch`	net472	2.6μs	0.691ns	2.49ns	0.161	1.02 KB
master	`CallElasticsearchAsync`	net6.0	1.37μs	0.548ns	2.05ns	0.0138	984 B
master	`CallElasticsearchAsync`	netcoreapp3.1	1.74μs	0.563ns	2.18ns	0.0139	1.06 KB
master	`CallElasticsearchAsync`	net472	2.7μs	1.83ns	7.07ns	0.17	1.08 KB
#5402	`CallElasticsearch`	net6.0	1.33μs	1.86ns	7.19ns	0.014	1.01 KB
#5402	`CallElasticsearch`	netcoreapp3.1	1.67μs	2.27ns	8.48ns	0.0132	1.01 KB
#5402	`CallElasticsearch`	net472	2.39μs	1.65ns	6.38ns	0.162	1.02 KB
#5402	`CallElasticsearchAsync`	net6.0	1.3μs	0.618ns	2.31ns	0.0136	984 B
#5402	`CallElasticsearchAsync`	netcoreapp3.1	1.73μs	1.66ns	6.22ns	0.0137	1.06 KB
#5402	`CallElasticsearchAsync`	net472	2.64μs	1.76ns	6.82ns	0.17	1.08 KB

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net6.0	1.32μs	0.573ns	2.14ns	0.0132	928 B
master	`ExecuteAsync`	netcoreapp3.1	1.76μs	1.46ns	5.46ns	0.0122	928 B
master	`ExecuteAsync`	net472	1.97μs	0.745ns	2.88ns	0.141	891 B
#5402	`ExecuteAsync`	net6.0	1.35μs	0.814ns	3.15ns	0.0128	928 B
#5402	`ExecuteAsync`	netcoreapp3.1	1.77μs	1.25ns	4.69ns	0.0124	928 B
#5402	`ExecuteAsync`	net472	1.94μs	0.667ns	2.58ns	0.141	891 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	4.19μs	1.07ns	4.13ns	0.0293	2.16 KB
master	`SendAsync`	netcoreapp3.1	5.1μs	1.55ns	5.81ns	0.0358	2.7 KB
master	`SendAsync`	net472	7.62μs	2.04ns	7.9ns	0.484	3.05 KB
#5402	`SendAsync`	net6.0	4.13μs	7.47ns	28ns	0.0304	2.16 KB
#5402	`SendAsync`	netcoreapp3.1	5.06μs	2.02ns	7.54ns	0.0355	2.7 KB
#5402	`SendAsync`	net472	7.7μs	3.02ns	11.7ns	0.483	3.05 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	1.66μs	1.03ns	3.98ns	0.0231	1.65 KB
master	`EnrichedLog`	netcoreapp3.1	2.18μs	3.87ns	15ns	0.0225	1.65 KB
master	`EnrichedLog`	net472	2.69μs	1.87ns	7.01ns	0.25	1.57 KB
#5402	`EnrichedLog`	net6.0	1.51μs	1.05ns	4.05ns	0.0234	1.65 KB
#5402	`EnrichedLog`	netcoreapp3.1	2.2μs	1.18ns	4.59ns	0.0218	1.65 KB
#5402	`EnrichedLog`	net472	2.69μs	2.22ns	8.32ns	0.25	1.57 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net6.0	112μs	139ns	539ns	0.0558	0	4.23 KB
master	`EnrichedLog`	netcoreapp3.1	117μs	162ns	626ns	0	0	4.23 KB
master	`EnrichedLog`	net472	145μs	62.5ns	242ns	0.653	0.218	4.41 KB
#5402	`EnrichedLog`	net6.0	111μs	66.7ns	240ns	0.0553	0	4.23 KB
#5402	`EnrichedLog`	netcoreapp3.1	119μs	153ns	592ns	0	0	4.23 KB
#5402	`EnrichedLog`	net472	146μs	79ns	306ns	0.655	0.218	4.41 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	3.08μs	1.1ns	4.25ns	0.0309	2.21 KB
master	`EnrichedLog`	netcoreapp3.1	4.24μs	3.02ns	11.7ns	0.0296	2.21 KB
master	`EnrichedLog`	net472	5μs	1.2ns	4.48ns	0.32	2.02 KB
#5402	`EnrichedLog`	net6.0	2.94μs	1.19ns	4.46ns	0.031	2.21 KB
#5402	`EnrichedLog`	netcoreapp3.1	4.24μs	2.27ns	8.8ns	0.0298	2.21 KB
#5402	`EnrichedLog`	net472	4.91μs	1.56ns	6.05ns	0.319	2.02 KB

Benchmarks.Trace.RedisBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #5402

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.RedisBenchmark.SendReceive‑net6.0	1.156	1,319.86	1,526.04

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendReceive`	net6.0	1.32μs	0.76ns	2.84ns	0.0165	1.18 KB
master	`SendReceive`	netcoreapp3.1	1.85μs	0.771ns	2.89ns	0.0157	1.18 KB
master	`SendReceive`	net472	2.25μs	1.67ns	6.45ns	0.187	1.18 KB
#5402	`SendReceive`	net6.0	1.53μs	0.702ns	2.63ns	0.0167	1.18 KB
#5402	`SendReceive`	netcoreapp3.1	1.84μs	0.606ns	2.27ns	0.0157	1.18 KB
#5402	`SendReceive`	net472	2.26μs	0.938ns	3.51ns	0.187	1.18 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.72μs	0.583ns	2.18ns	0.0218	1.55 KB
master	`EnrichedLog`	netcoreapp3.1	3.91μs	1.77ns	6.37ns	0.0215	1.6 KB
master	`EnrichedLog`	net472	4.38μs	1.98ns	7.15ns	0.315	1.99 KB
#5402	`EnrichedLog`	net6.0	2.72μs	1.17ns	4.39ns	0.0219	1.55 KB
#5402	`EnrichedLog`	netcoreapp3.1	4μs	1.2ns	4.32ns	0.0218	1.6 KB
#5402	`EnrichedLog`	net472	4.3μs	1.39ns	5.4ns	0.315	1.99 KB

Benchmarks.Trace.SpanBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	522ns	0.579ns	2.24ns	0.00757	552 B
master	`StartFinishSpan`	netcoreapp3.1	685ns	1.01ns	3.92ns	0.00759	552 B
master	`StartFinishSpan`	net472	746ns	1.47ns	5.7ns	0.0877	554 B
master	`StartFinishScope`	net6.0	546ns	0.706ns	2.64ns	0.00931	672 B
master	`StartFinishScope`	netcoreapp3.1	908ns	1.24ns	4.66ns	0.0092	672 B
master	`StartFinishScope`	net472	900ns	1.73ns	6.69ns	0.101	634 B
#5402	`StartFinishSpan`	net6.0	578ns	0.923ns	3.57ns	0.00778	552 B
#5402	`StartFinishSpan`	netcoreapp3.1	679ns	1.07ns	4.14ns	0.00741	552 B
#5402	`StartFinishSpan`	net472	737ns	1.7ns	6.37ns	0.0877	554 B
#5402	`StartFinishScope`	net6.0	593ns	1.35ns	5.22ns	0.00928	672 B
#5402	`StartFinishScope`	netcoreapp3.1	856ns	1.37ns	5.32ns	0.0089	672 B
#5402	`StartFinishScope`	net472	918ns	1.68ns	6.5ns	0.101	634 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	644ns	0.935ns	3.62ns	0.00945	672 B
master	`RunOnMethodBegin`	netcoreapp3.1	927ns	1.07ns	4.16ns	0.00897	672 B
master	`RunOnMethodBegin`	net472	1.12μs	1.99ns	7.7ns	0.1	634 B
#5402	`RunOnMethodBegin`	net6.0	587ns	0.893ns	3.46ns	0.00946	672 B
#5402	`RunOnMethodBegin`	netcoreapp3.1	1.03μs	1.94ns	7.5ns	0.00925	672 B
#5402	`RunOnMethodBegin`	net472	1.1μs	1.65ns	6.39ns	0.101	634 B

…tion test project This was originally added because I couldn't get the Razor Pages unit tests to work directly. I don't know why, because they _do_ work now

bouwkast

👍

andrewlock added area:tests unit tests, integration tests area:test-apps apps used to test integrations labels Apr 4, 2024

andrewlock requested review from a team as code owners April 4, 2024 15:24

lucaspimentel approved these changes Apr 4, 2024

View reviewed changes

lucaspimentel requested a review from a team April 4, 2024 16:15

andrewlock force-pushed the andrew/ci/remove-direct-razor-pages-reference branch from 6c8b5c4 to 9aef0fe Compare April 4, 2024 19:37

andrewlock added 4 commits April 5, 2024 08:03

Remove direct reference to Samples.AspNetCore.RazorPages from integra…

1710ea5

…tion test project This was originally added because I couldn't get the Razor Pages unit tests to work directly. I don't know why, because they _do_ work now

Remove the special-casing for Samples.AspNetCore.RazorPages

2036b83

Move the pages to the AspNetCoreRazorPages subfolder

95d75f0

Don't run against .NET Core 2.1

fa42d01

andrewlock force-pushed the andrew/ci/remove-direct-razor-pages-reference branch from 9aef0fe to fa42d01 Compare April 5, 2024 07:18

bouwkast approved these changes Apr 8, 2024

View reviewed changes

andrewlock merged commit 5276931 into master Apr 9, 2024
52 of 57 checks passed

andrewlock deleted the andrew/ci/remove-direct-razor-pages-reference branch April 9, 2024 08:45

github-actions bot added this to the vNext-v2 milestone Apr 9, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove direct reference to Samples.AspNetCore.RazorPages from integration test project #5402

Remove direct reference to Samples.AspNetCore.RazorPages from integration test project #5402

andrewlock commented Apr 4, 2024

datadog-ddstaging bot commented Apr 4, 2024 •

edited

Loading

lucaspimentel commented Apr 4, 2024 •

edited

Loading

lucaspimentel left a comment

andrewlock commented Apr 4, 2024 •

edited

Loading

andrewlock commented Apr 4, 2024 •

edited

Loading

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #5402

Raw results

Raw results

Raw results

Raw results

bouwkast left a comment

Remove direct reference to Samples.AspNetCore.RazorPages from integration test project #5402

Remove direct reference to Samples.AspNetCore.RazorPages from integration test project #5402

Conversation

andrewlock commented Apr 4, 2024

Summary of changes

Reason for change

Implementation details

Test coverage

Other details

datadog-ddstaging bot commented Apr 4, 2024 • edited Loading

Datadog Report

lucaspimentel commented Apr 4, 2024 • edited Loading

lucaspimentel left a comment

Choose a reason for hiding this comment

andrewlock commented Apr 4, 2024 • edited Loading

Execution-Time Benchmarks Report ⏱️

andrewlock commented Apr 4, 2024 • edited Loading

Benchmarks Report for tracer 🐌

Benchmark details

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #5402

Raw results

Raw results

Raw results

Raw results

bouwkast left a comment

Choose a reason for hiding this comment

datadog-ddstaging bot commented Apr 4, 2024 •

edited

Loading

lucaspimentel commented Apr 4, 2024 •

edited

Loading

andrewlock commented Apr 4, 2024 •

edited

Loading

andrewlock commented Apr 4, 2024 •

edited

Loading