[Tracer] scrub `_dd.agent_psr` from test snapshots #5562

lucaspimentel · 2024-05-14T21:24:32Z

Summary of changes

Remove _dd.agent_psr from more test snapshots. Leftover from #5545.

edit: TraceAnnotationsVersionMismatchNewerNuGetTests keeps being flaky (sometimes it adds the tag, I assume when it loads an older version of the tracer), so I'm scrubbing _dd.agent_psr out of all snapshots.

Reason for change

Snapshots tests that normally only run on master are failing after merging #5545.

Implementation details

Update the snapshot files.

edit: and update the default Verifier settings to remove _dd.agent_psr

Test coverage

Exactly.

Other details

github-actions · 2024-05-14T21:25:18Z

Snapshots difference summary

The following differences have been observed in committed snapshots. It is meant to help the reviewer.
The diff is simplistic, so please check some files anyway while we improve it.

1856 occurrences of :

-      _dd.agent_psr: 1.0,

datadog-ddstaging · 2024-05-14T21:41:46Z

Datadog Report

Branch report: lpimentel/agent-sampling-rate-tag-redux
Commit report: b97eddd
Test service: dd-trace-dotnet

❌ 3 Failed (0 Known Flaky), 337299 Passed, 1565 Skipped, 13h 29m 37.62s Total Time

❌ Failed Tests (3)

SubmitsTraces - Datadog.Trace.ClrProfiler.IntegrationTests.AWS.AwsLambdaTests - Details
Expand for error
```
 Value cannot be null. (Parameter 'source')
```
SubmitsTraces - Datadog.Trace.ClrProfiler.IntegrationTests.AWS.AwsLambdaTests - Details
Expand for error
```
 Value cannot be null. (Parameter 'source')
```
SubmitsTraces - Datadog.Trace.ClrProfiler.IntegrationTests.AWS.AwsLambdaTests - Details
Expand for error
```
 Value cannot be null. (Parameter 'source')
```

andrewlock · 2024-05-14T21:47:22Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5562) - mean (74ms)  : 65, 83
     .   : milestone, 74,
    master - mean (75ms)  : 67, 83
     .   : milestone, 75,

    section CallTarget+Inlining+NGEN
    This PR (5562) - mean (1,004ms)  : 971, 1038
     .   : milestone, 1004,
    master - mean (1,013ms)  : 987, 1039
     .   : milestone, 1013,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5562) - mean (110ms)  : 107, 114
     .   : milestone, 110,
    master - mean (111ms)  : 108, 115
     .   : milestone, 111,

    section CallTarget+Inlining+NGEN
    This PR (5562) - mean (730ms)  : 704, 756
     .   : milestone, 730,
    master - mean (729ms)  : 708, 750
     .   : milestone, 729,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5562) - mean (95ms)  : 91, 99
     .   : milestone, 95,
    master - mean (94ms)  : 90, 97
     .   : milestone, 94,

    section CallTarget+Inlining+NGEN
    This PR (5562) - mean (680ms)  : 660, 700
     .   : milestone, 680,
    master - mean (679ms)  : 655, 703
     .   : milestone, 679,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5562) - mean (192ms)  : 189, 195
     .   : milestone, 192,
    master - mean (193ms)  : 189, 196
     .   : milestone, 193,

    section CallTarget+Inlining+NGEN
    This PR (5562) - mean (1,109ms)  : 1093, 1124
     .   : milestone, 1109,
    master - mean (1,115ms)  : 1091, 1139
     .   : milestone, 1115,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5562) - mean (275ms)  : 270, 279
     .   : milestone, 275,
    master - mean (276ms)  : 270, 281
     .   : milestone, 276,

    section CallTarget+Inlining+NGEN
    This PR (5562) - mean (915ms)  : 898, 932
     .   : milestone, 915,
    master - mean (913ms)  : 891, 935
     .   : milestone, 913,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5562) - mean (264ms)  : 260, 269
     .   : milestone, 264,
    master - mean (265ms)  : 261, 269
     .   : milestone, 265,

    section CallTarget+Inlining+NGEN
    This PR (5562) - mean (891ms)  : 871, 911
     .   : milestone, 891,
    master - mean (895ms)  : 873, 917
     .   : milestone, 895,

andrewlock · 2024-05-14T22:00:23Z

Benchmarks Report for tracer 🐌

Benchmarks for #5562 compared to master:

1 benchmarks are slower, with geometric mean 1.168
All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	8.8μs	48.6ns	315ns	0.0218	0.00873	0	7.56 KB
master	`StartStopWithChild`	netcoreapp3.1	10.6μs	49.8ns	193ns	0.0215	0.0107	0	7.66 KB
master	`StartStopWithChild`	net472	17.1μs	56.2ns	218ns	1.33	0.342	0.102	8.06 KB
#5562	`StartStopWithChild`	net6.0	8.6μs	47.8ns	287ns	0.0295	0.0169	0.00421	7.55 KB
#5562	`StartStopWithChild`	netcoreapp3.1	11.2μs	60.9ns	339ns	0.0213	0.0106	0	7.65 KB
#5562	`StartStopWithChild`	net472	17μs	54.6ns	211ns	1.34	0.361	0.103	8.05 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	507μs	168ns	629ns	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	622μs	459ns	1.78μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	823μs	404ns	1.51μs	0.411	3.3 KB
#5562	`WriteAndFlushEnrichedTraces`	net6.0	468μs	121ns	469ns	0	2.7 KB
#5562	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	636μs	311ns	1.16μs	0	2.7 KB
#5562	`WriteAndFlushEnrichedTraces`	net472	819μs	303ns	1.13μs	0.411	3.3 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net6.0	170μs	95ns	355ns	0.169	18.49 KB
master	`SendRequest`	netcoreapp3.1	192μs	270ns	1.05μs	0.192	20.65 KB
master	`SendRequest`	net472	0.000792ns	0.00041ns	0.00153ns	0	0 b
#5562	`SendRequest`	net6.0	172μs	176ns	636ns	0.171	18.49 KB
#5562	`SendRequest`	netcoreapp3.1	192μs	188ns	702ns	0.191	20.65 KB
#5562	`SendRequest`	net472	0.00133ns	0.000381ns	0.00143ns	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	547μs	301ns	1.17μs	0.548	0	0	41.71 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	650μs	2.49μs	9.66μs	0.324	0	0	41.69 KB
master	`WriteAndFlushEnrichedTraces`	net472	818μs	3.69μs	14.3μs	8.22	2.47	0.411	53.24 KB
#5562	`WriteAndFlushEnrichedTraces`	net6.0	558μs	357ns	1.38μs	0.541	0	0	41.65 KB
#5562	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	658μs	2.33μs	9.03μs	0.336	0	0	41.76 KB
#5562	`WriteAndFlushEnrichedTraces`	net472	836μs	3.52μs	13.6μs	8.31	2.37	0.396	53.28 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteNonQuery`	net6.0	1.09μs	0.739ns	2.86ns	0.0114	808 B
master	`ExecuteNonQuery`	netcoreapp3.1	1.5μs	0.948ns	3.67ns	0.0111	808 B
master	`ExecuteNonQuery`	net472	1.77μs	0.424ns	1.47ns	0.122	770 B
#5562	`ExecuteNonQuery`	net6.0	1.08μs	0.398ns	1.54ns	0.0113	808 B
#5562	`ExecuteNonQuery`	netcoreapp3.1	1.51μs	1.34ns	5.19ns	0.0105	808 B
#5562	`ExecuteNonQuery`	net472	1.74μs	0.659ns	2.55ns	0.122	770 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`CallElasticsearch`	net6.0	1.25μs	0.668ns	2.59ns	0.0143	1.03 KB
master	`CallElasticsearch`	netcoreapp3.1	1.65μs	1.63ns	6.31ns	0.0141	1.03 KB
master	`CallElasticsearch`	net472	2.62μs	0.951ns	3.68ns	0.166	1.04 KB
master	`CallElasticsearchAsync`	net6.0	1.28μs	0.6ns	2.32ns	0.0141	1.01 KB
master	`CallElasticsearchAsync`	netcoreapp3.1	1.7μs	1.31ns	5.06ns	0.0145	1.08 KB
master	`CallElasticsearchAsync`	net472	2.55μs	0.849ns	3.29ns	0.175	1.1 KB
#5562	`CallElasticsearch`	net6.0	1.34μs	0.459ns	1.65ns	0.0142	1.03 KB
#5562	`CallElasticsearch`	netcoreapp3.1	1.61μs	1.55ns	5.79ns	0.0135	1.03 KB
#5562	`CallElasticsearch`	net472	2.43μs	1.38ns	5.17ns	0.165	1.04 KB
#5562	`CallElasticsearchAsync`	net6.0	1.33μs	0.463ns	1.67ns	0.014	1.01 KB
#5562	`CallElasticsearchAsync`	netcoreapp3.1	1.76μs	1.05ns	4.05ns	0.0149	1.08 KB
#5562	`CallElasticsearchAsync`	net472	2.66μs	2.96ns	11.1ns	0.175	1.1 KB

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`ExecuteAsync`	net6.0	1.28μs	0.925ns	3.46ns	0.0134	0	952 B
master	`ExecuteAsync`	netcoreapp3.1	1.64μs	0.676ns	2.62ns	0.0124	0	952 B
master	`ExecuteAsync`	net472	1.77μs	0.66ns	2.56ns	0.145	0.000883	915 B
#5562	`ExecuteAsync`	net6.0	1.31μs	1.15ns	4.44ns	0.0131	0	952 B
#5562	`ExecuteAsync`	netcoreapp3.1	1.73μs	1.62ns	6.08ns	0.0121	0	952 B
#5562	`ExecuteAsync`	net472	1.87μs	0.813ns	3.04ns	0.145	0	915 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	4.11μs	1.74ns	6.75ns	0.0307	2.27 KB
master	`SendAsync`	netcoreapp3.1	5.08μs	3.47ns	12.5ns	0.0381	2.81 KB
master	`SendAsync`	net472	7.64μs	2.72ns	10.5ns	0.503	3.18 KB
#5562	`SendAsync`	net6.0	4.27μs	0.952ns	3.56ns	0.0319	2.27 KB
#5562	`SendAsync`	netcoreapp3.1	5.27μs	4.98ns	18ns	0.0369	2.81 KB
#5562	`SendAsync`	net472	7.76μs	3.25ns	11.7ns	0.504	3.18 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	1.58μs	1.08ns	4.03ns	0.0238	1.7 KB
master	`EnrichedLog`	netcoreapp3.1	2.23μs	1.18ns	4.41ns	0.0235	1.7 KB
master	`EnrichedLog`	net472	2.69μs	1.62ns	6.26ns	0.257	1.62 KB
#5562	`EnrichedLog`	net6.0	1.47μs	0.694ns	2.5ns	0.0239	1.7 KB
#5562	`EnrichedLog`	netcoreapp3.1	2.13μs	1.42ns	5.33ns	0.0224	1.7 KB
#5562	`EnrichedLog`	net472	2.65μs	1.59ns	6.15ns	0.257	1.62 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net6.0	113μs	86.4ns	323ns	0.0562	0	4.28 KB
master	`EnrichedLog`	netcoreapp3.1	117μs	119ns	459ns	0.0586	0	4.28 KB
master	`EnrichedLog`	net472	147μs	50.9ns	190ns	0.659	0.22	4.46 KB
#5562	`EnrichedLog`	net6.0	113μs	74.9ns	290ns	0	0	4.28 KB
#5562	`EnrichedLog`	netcoreapp3.1	117μs	153ns	591ns	0	0	4.28 KB
#5562	`EnrichedLog`	net472	148μs	68.5ns	237ns	0.667	0.222	4.46 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.88μs	0.727ns	2.72ns	0.0306	2.25 KB
master	`EnrichedLog`	netcoreapp3.1	4.25μs	1.88ns	7.04ns	0.0299	2.25 KB
master	`EnrichedLog`	net472	4.88μs	1.54ns	5.78ns	0.328	2.07 KB
#5562	`EnrichedLog`	net6.0	3.04μs	16.1ns	85.4ns	0.0313	2.25 KB
#5562	`EnrichedLog`	netcoreapp3.1	4.15μs	1.22ns	4.56ns	0.0309	2.25 KB
#5562	`EnrichedLog`	net472	4.89μs	1.26ns	4.89ns	0.328	2.07 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`SendReceive`	net6.0	1.35μs	0.964ns	3.61ns	0.0168	0	1.2 KB
master	`SendReceive`	netcoreapp3.1	1.82μs	3.32ns	12.4ns	0.0163	0	1.2 KB
master	`SendReceive`	net472	2.25μs	1.42ns	5.33ns	0.19	0.00113	1.2 KB
#5562	`SendReceive`	net6.0	1.33μs	0.834ns	3.23ns	0.0165	0	1.2 KB
#5562	`SendReceive`	netcoreapp3.1	1.86μs	1.07ns	4ns	0.0167	0	1.2 KB
#5562	`SendReceive`	net472	2.26μs	0.966ns	3.74ns	0.191	0	1.2 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.88μs	3.25ns	12.2ns	0.0216	1.6 KB
master	`EnrichedLog`	netcoreapp3.1	4.18μs	4.84ns	18.8ns	0.0223	1.65 KB
master	`EnrichedLog`	net472	4.44μs	1.57ns	6.09ns	0.322	2.04 KB
#5562	`EnrichedLog`	net6.0	2.73μs	1.45ns	5.6ns	0.0218	1.6 KB
#5562	`EnrichedLog`	netcoreapp3.1	3.94μs	2.09ns	8.11ns	0.0217	1.65 KB
#5562	`EnrichedLog`	net472	4.52μs	2.14ns	8.01ns	0.324	2.04 KB

Benchmarks.Trace.SpanBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	465ns	0.144ns	0.558ns	0.00814	576 B
master	`StartFinishSpan`	netcoreapp3.1	736ns	2.96ns	11.5ns	0.00758	576 B
master	`StartFinishSpan`	net472	892ns	0.204ns	0.764ns	0.0914	578 B
master	`StartFinishScope`	net6.0	626ns	0.293ns	1.1ns	0.0098	696 B
master	`StartFinishScope`	netcoreapp3.1	906ns	1.76ns	6.8ns	0.00952	696 B
master	`StartFinishScope`	net472	1.07μs	0.694ns	2.69ns	0.104	658 B
#5562	`StartFinishSpan`	net6.0	474ns	0.122ns	0.455ns	0.00807	576 B
#5562	`StartFinishSpan`	netcoreapp3.1	682ns	3.78ns	23.3ns	0.0078	576 B
#5562	`StartFinishSpan`	net472	860ns	0.742ns	2.88ns	0.0914	578 B
#5562	`StartFinishScope`	net6.0	617ns	0.149ns	0.558ns	0.00967	696 B
#5562	`StartFinishScope`	netcoreapp3.1	841ns	0.223ns	0.805ns	0.00931	696 B
#5562	`StartFinishScope`	net472	1.02μs	0.755ns	2.92ns	0.104	658 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #5562

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin‑net6.0	1.168	606.79	708.81

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	607ns	0.185ns	0.715ns	0.00994	696 B
master	`RunOnMethodBegin`	netcoreapp3.1	931ns	1.33ns	5.14ns	0.00914	696 B
master	`RunOnMethodBegin`	net472	1.13μs	0.343ns	1.33ns	0.104	658 B
#5562	`RunOnMethodBegin`	net6.0	709ns	0.726ns	2.81ns	0.00973	696 B
#5562	`RunOnMethodBegin`	netcoreapp3.1	983ns	0.399ns	1.55ns	0.00945	696 B
#5562	`RunOnMethodBegin`	net472	1.17μs	0.871ns	3.37ns	0.104	658 B

andrewlock · 2024-05-14T22:11:32Z

Throughput/Crank Report:zap:

Throughput results for AspNetCoreSimpleController comparing the following branches/commits:

Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.

Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!

gantt
    title Throughput Linux x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5562) (11.992M)   : 0, 11992341
    master (11.926M)   : 0, 11926098
    benchmarks/2.9.0 (11.817M)   : 0, 11816893

    section Automatic
    This PR (5562) (7.994M)   : 0, 7993685
    master (8.024M)   : 0, 8023608
    benchmarks/2.9.0 (8.480M)   : 0, 8480079

    section Trace stats
    master (8.370M)   : 0, 8370281

    section Manual
    This PR (5562) (10.251M)   : 0, 10250542
    master (10.123M)   : 0, 10123133

    section Manual + Automatic
    This PR (5562) (7.629M)   : 0, 7629198
    master (7.618M)   : 0, 7617662

    section Version Conflict
    master (6.782M)   : 0, 6782273

gantt
    title Throughput Linux arm64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5562) (9.531M)   : 0, 9530577
    master (9.727M)   : 0, 9726736
    benchmarks/2.9.0 (9.598M)   : 0, 9597540

    section Automatic
    This PR (5562) (6.629M)   : 0, 6628874
    master (6.648M)   : 0, 6647512

    section Trace stats
    master (6.936M)   : 0, 6935628

    section Manual
    This PR (5562) (8.388M)   : 0, 8388333
    master (8.392M)   : 0, 8392466

    section Manual + Automatic
    This PR (5562) (6.263M)   : 0, 6262996
    master (6.156M)   : 0, 6155670

    section Version Conflict
    master (5.584M)   : 0, 5584079

gantt
    title Throughput Windows x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5562) (9.860M)   : 0, 9860184
    master (9.600M)   : 0, 9599975
    benchmarks/2.9.0 (10.041M)   : 0, 10040980

    section Automatic
    This PR (5562) (6.935M)   : 0, 6935380
    master (7.018M)   : 0, 7018195
    benchmarks/2.9.0 (7.137M)   : 0, 7137430

    section Trace stats
    master (7.433M)   : 0, 7432726

    section Manual
    This PR (5562) (8.592M)   : 0, 8592335
    master (8.616M)   : 0, 8615941

    section Manual + Automatic
    This PR (5562) (6.697M)   : 0, 6697167
    master (6.780M)   : 0, 6780288

    section Version Conflict
    master (6.086M)   : 0, 6086434

andrewlock · 2024-05-16T03:57:31Z

Benchmarks Report for appsec 🐌

Benchmarks for #5562 compared to master:

All benchmarks have the same speed
2 benchmarks have fewer allocations
1 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.Asm.AppSecBodyBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`AllCycleSimpleBody`	net6.0	71.6μs	92.8ns	359ns	0.0715	0	6 KB
master	`AllCycleSimpleBody`	netcoreapp3.1	63.4μs	68.8ns	248ns	0.0946	0	6.94 KB
master	`AllCycleSimpleBody`	net472	47.9μs	46.2ns	173ns	1.3	0	8.32 KB
master	`AllCycleMoreComplexBody`	net6.0	78.4μs	87.5ns	315ns	0.119	0	9.5 KB
master	`AllCycleMoreComplexBody`	netcoreapp3.1	69.5μs	99ns	383ns	0.139	0	10.36 KB
master	`AllCycleMoreComplexBody`	net472	54.7μs	46.8ns	175ns	1.88	0.0272	11.84 KB
master	`ObjectExtractorSimpleBody`	net6.0	144ns	0.217ns	0.841ns	0.00398	0	280 B
master	`ObjectExtractorSimpleBody`	netcoreapp3.1	229ns	0.214ns	0.801ns	0.0037	0	272 B
master	`ObjectExtractorSimpleBody`	net472	170ns	0.233ns	0.87ns	0.0446	0	281 B
master	`ObjectExtractorMoreComplexBody`	net6.0	3.1μs	1.38ns	5.35ns	0.0524	0	3.78 KB
master	`ObjectExtractorMoreComplexBody`	netcoreapp3.1	4.02μs	1.56ns	6.05ns	0.0498	0	3.69 KB
master	`ObjectExtractorMoreComplexBody`	net472	3.77μs	2.32ns	8.67ns	0.602	0.00566	3.8 KB
#5562	`AllCycleSimpleBody`	net6.0	72.3μs	118ns	456ns	0.0727	0	6 KB
#5562	`AllCycleSimpleBody`	netcoreapp3.1	61.3μs	113ns	438ns	0.0922	0	6.94 KB
#5562	`AllCycleSimpleBody`	net472	49.7μs	283ns	2.06μs	1.32	0	8.33 KB
#5562	`AllCycleMoreComplexBody`	net6.0	78μs	110ns	426ns	0.117	0	9.5 KB
#5562	`AllCycleMoreComplexBody`	netcoreapp3.1	68.9μs	103ns	398ns	0.139	0	10.36 KB
#5562	`AllCycleMoreComplexBody`	net472	55μs	63.1ns	244ns	1.86	0.0274	11.84 KB
#5562	`ObjectExtractorSimpleBody`	net6.0	137ns	0.136ns	0.507ns	0.00397	0	280 B
#5562	`ObjectExtractorSimpleBody`	netcoreapp3.1	210ns	0.458ns	1.77ns	0.00376	0	272 B
#5562	`ObjectExtractorSimpleBody`	net472	174ns	0.223ns	0.863ns	0.0446	0	281 B
#5562	`ObjectExtractorMoreComplexBody`	net6.0	3.15μs	3.28ns	12.3ns	0.0524	0	3.78 KB
#5562	`ObjectExtractorMoreComplexBody`	netcoreapp3.1	4.05μs	2.41ns	9.03ns	0.0507	0	3.69 KB
#5562	`ObjectExtractorMoreComplexBody`	net472	3.77μs	6.3ns	24.4ns	0.602	0.00563	3.8 KB

Benchmarks.Trace.Asm.AppSecEncoderBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EncodeArgs`	net6.0	41.1μs	16.5ns	59.5ns	0.454	0	32.4 KB
master	`EncodeArgs`	netcoreapp3.1	55.3μs	26.2ns	98ns	0.44	0	32.4 KB
master	`EncodeArgs`	net472	68.9μs	38.7ns	150ns	5.16	0.0688	32.5 KB
master	`EncodeLegacyArgs`	net6.0	71.1μs	39.7ns	154ns	0	0	2.14 KB
master	`EncodeLegacyArgs`	netcoreapp3.1	105μs	131ns	508ns	0	0	2.15 KB
master	`EncodeLegacyArgs`	net472	151μs	62.1ns	224ns	0.305	0	2.15 KB
#5562	`EncodeArgs`	net6.0	40.8μs	18.9ns	73.2ns	0.451	0	32.4 KB
#5562	`EncodeArgs`	netcoreapp3.1	55μs	25.5ns	98.7ns	0.438	0	32.4 KB
#5562	`EncodeArgs`	net472	69μs	46ns	178ns	5.14	0.069	32.5 KB
#5562	`EncodeLegacyArgs`	net6.0	73.9μs	49.7ns	192ns	0.0367	0	2.14 KB
#5562	`EncodeLegacyArgs`	netcoreapp3.1	104μs	57.1ns	221ns	0	0	2.15 KB
#5562	`EncodeLegacyArgs`	net472	154μs	58.2ns	226ns	0.308	0	2.15 KB

Benchmarks.Trace.Asm.AppSecWafBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunWafRealisticBenchmark`	net6.0	181μs	82.3ns	319ns	0	2.34 KB
master	`RunWafRealisticBenchmark`	netcoreapp3.1	207μs	1.14μs	7.57μs	0	2.3 KB
master	`RunWafRealisticBenchmark`	net472	215μs	99.1ns	371ns	0.322	2.36 KB
master	`RunWafRealisticBenchmarkWithAttack`	net6.0	132μs	769ns	6.61μs	0	1.44 KB
master	`RunWafRealisticBenchmarkWithAttack`	netcoreapp3.1	131μs	160ns	620ns	0	1.42 KB
master	`RunWafRealisticBenchmarkWithAttack`	net472	141μs	50.6ns	196ns	0.212	1.45 KB
#5562	`RunWafRealisticBenchmark`	net6.0	183μs	126ns	487ns	0	2.35 KB
#5562	`RunWafRealisticBenchmark`	netcoreapp3.1	198μs	283ns	1.06μs	0	2.3 KB
#5562	`RunWafRealisticBenchmark`	net472	216μs	123ns	478ns	0.324	2.36 KB
#5562	`RunWafRealisticBenchmarkWithAttack`	net6.0	122μs	91.8ns	356ns	0	1.44 KB
#5562	`RunWafRealisticBenchmarkWithAttack`	netcoreapp3.1	130μs	185ns	716ns	0	1.42 KB
#5562	`RunWafRealisticBenchmarkWithAttack`	net472	142μs	50ns	194ns	0.212	1.45 KB

Benchmarks.Trace.Iast.StringAspectsBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #5562

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑net6.0	204.04 KB	205.2 KB	1.16 KB	0.57%

Fewer allocations 🎉 in #5562

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark‑net472	61.9 KB	59.36 KB	-2.54 KB	-4.10%
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑netcoreapp3.1	213.86 KB	203.1 KB	-10.77 KB	-5.03%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Allocated
master	`StringConcatBenchmark`	net6.0	51.9μs	231ns	866ns	43.44 KB
master	`StringConcatBenchmark`	netcoreapp3.1	53μs	128ns	442ns	42.64 KB
master	`StringConcatBenchmark`	net472	38.1μs	99ns	357ns	61.9 KB
master	`StringConcatAspectBenchmark`	net6.0	286μs	6.15μs	59.6μs	204.04 KB
master	`StringConcatAspectBenchmark`	netcoreapp3.1	320μs	1.57μs	6.27μs	213.86 KB
master	`StringConcatAspectBenchmark`	net472	245μs	1.05μs	3.79μs	221.18 KB
#5562	`StringConcatBenchmark`	net6.0	52.1μs	220ns	794ns	43.44 KB
#5562	`StringConcatBenchmark`	netcoreapp3.1	60.2μs	762ns	7.38μs	42.64 KB
#5562	`StringConcatBenchmark`	net472	38.3μs	94.7ns	342ns	59.36 KB
#5562	`StringConcatAspectBenchmark`	net6.0	295μs	1.64μs	10.8μs	205.2 KB
#5562	`StringConcatAspectBenchmark`	netcoreapp3.1	300μs	1.96μs	19.1μs	203.1 KB
#5562	`StringConcatAspectBenchmark`	net472	235μs	1.2μs	5.48μs	221.18 KB

robertpi · 2024-05-16T07:26:43Z

Nice.

Also nice.

lucaspimentel added the area:tests unit tests, integration tests label May 14, 2024

lucaspimentel requested review from a team as code owners May 14, 2024 21:24

andrewlock approved these changes May 15, 2024

View reviewed changes

lucaspimentel changed the title ~~Remove _dd.agent_psr from more test snapshots~~ [Tracer] scrub _dd.agent_psr from test snapshots May 15, 2024

lucaspimentel force-pushed the lpimentel/agent-sampling-rate-tag-redux branch 2 times, most recently from e8cc486 to b222d61 Compare May 16, 2024 02:43

robertpi approved these changes May 16, 2024

View reviewed changes

lucaspimentel force-pushed the lpimentel/agent-sampling-rate-tag-redux branch from b222d61 to e2c2881 Compare May 16, 2024 11:48

lucaspimentel added 11 commits May 16, 2024 07:52

update test snapshots (ASP.NET Core MVC 3.0)

3e9d1c3

update test snapshots

baba1cc

clean up and rename

e38298b

re-order parameters

fae64f0

make parameters non-nullable

0de2d8f

rename methods

c282924

to expression lambda

a30a8fc

reorder methods

3675df4

scrub "_dd.agent_psr"

dc9009c

revert rename

c81c6f7

fix compiler error

b97eddd

lucaspimentel force-pushed the lpimentel/agent-sampling-rate-tag-redux branch from e2c2881 to b97eddd Compare May 16, 2024 11:52

lucaspimentel merged commit 823f470 into master May 16, 2024
55 of 58 checks passed

lucaspimentel deleted the lpimentel/agent-sampling-rate-tag-redux branch May 16, 2024 13:26

github-actions bot added this to the vNext-v2 milestone May 16, 2024

andrewlock mentioned this pull request May 16, 2024

Failing serverless tests are not reported in CI #5574

Merged

This was referenced May 16, 2024

[Tests] add null checks in VerifyHelper #5575

Merged

[Tracer][Tests] scrub _dd.agent_psr tag from TraceAnnotationsTests snapshots only #5591

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Tracer] scrub `_dd.agent_psr` from test snapshots #5562

[Tracer] scrub `_dd.agent_psr` from test snapshots #5562

lucaspimentel commented May 14, 2024 •

edited

Loading

github-actions bot commented May 14, 2024

datadog-ddstaging bot commented May 14, 2024 •

edited

Loading

andrewlock commented May 14, 2024 •

edited

Loading

andrewlock commented May 14, 2024 •

edited

Loading

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #5562

Raw results

andrewlock commented May 14, 2024 •

edited

Loading

andrewlock commented May 16, 2024

Raw results

Raw results

Raw results

More allocations ⚠️ in #5562

Fewer allocations 🎉 in #5562

Raw results

robertpi commented May 16, 2024

[Tracer] scrub _dd.agent_psr from test snapshots #5562

[Tracer] scrub _dd.agent_psr from test snapshots #5562

Conversation

lucaspimentel commented May 14, 2024 • edited Loading

Summary of changes

Reason for change

Implementation details

Test coverage

Other details

github-actions bot commented May 14, 2024

Snapshots difference summary

datadog-ddstaging bot commented May 14, 2024 • edited Loading

Datadog Report

❌ Failed Tests (3)

andrewlock commented May 14, 2024 • edited Loading

Execution-Time Benchmarks Report ⏱️

andrewlock commented May 14, 2024 • edited Loading

Benchmarks Report for tracer 🐌

Benchmark details

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #5562

Raw results

andrewlock commented May 14, 2024 • edited Loading

Throughput/Crank Report:zap:

andrewlock commented May 16, 2024

Benchmarks Report for appsec 🐌

Benchmark details

Raw results

Raw results

Raw results

More allocations ⚠️ in #5562

Fewer allocations 🎉 in #5562

Raw results

robertpi commented May 16, 2024

[Tracer] scrub `_dd.agent_psr` from test snapshots #5562

[Tracer] scrub `_dd.agent_psr` from test snapshots #5562

lucaspimentel commented May 14, 2024 •

edited

Loading

datadog-ddstaging bot commented May 14, 2024 •

edited

Loading

andrewlock commented May 14, 2024 •

edited

Loading

andrewlock commented May 14, 2024 •

edited

Loading

andrewlock commented May 14, 2024 •

edited

Loading