Add attribute for skipping tests in CI without using `[Fact(Skip = "")]` #5756

andrewlock · 2024-07-01T09:54:04Z

Summary of changes

Adds a new trait, SkipInCI that can be used to skip tests in CI

Reason for change

We currently use the built-in [Fact(Skip="..."]) or [Theory(Skip="..."]) to skip tests that we don't want to run in CI. The downside to this is if you want to run a test locally, you have to explicitly change the code to remove the Skip, run the test, and remember to add it back later.

Instead, we can change the default Filter used by Nuke for each stage to be SkipInCI!=True. That way the tests are still excluded from CI (or if you run the "default" test suite locally), but you can still easily run the test without making code modifications if you want to run it directly from VS/Rider, or using nuke RunWindowsIntegrationTests -Filter SomeSkippedTest (for example).

Implementation details

Replaced existing usages of

[Fact(Skip="Some reason"])

with

[Trait("SkipInCI", "True")] // Some reason

and changed the default filters used by Nuke to exclude tests with this trait

Test coverage

This PR is the test - if it passes, we're good.

Other details

It would be nice to have the skip reason as the value of the trait, but the dotnet test syntax doesn't make it easy to select just based on the key, it's Key=Value, so we're stuck with comments.

vandonr

not too sure what to think of the code changes as it's an area I really don't know well, but I like the idea :)
I'll put my approval to contribute half of the required reviews ;)

andrewlock · 2024-07-01T10:47:19Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5756) - mean (74ms)  : 65, 82
     .   : milestone, 74,
    master - mean (73ms)  : 64, 81
     .   : milestone, 73,

    section CallTarget+Inlining+NGEN
    This PR (5756) - mean (902ms)  : 874, 930
     .   : milestone, 902,
    master - mean (896ms)  : 868, 924
     .   : milestone, 896,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5756) - mean (110ms)  : 106, 114
     .   : milestone, 110,
    master - mean (109ms)  : 106, 112
     .   : milestone, 109,

    section CallTarget+Inlining+NGEN
    This PR (5756) - mean (634ms)  : 618, 651
     .   : milestone, 634,
    master - mean (632ms)  : 617, 648
     .   : milestone, 632,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5756) - mean (95ms)  : 92, 99
     .   : milestone, 95,
    master - mean (93ms)  : 89, 97
     .   : milestone, 93,

    section CallTarget+Inlining+NGEN
    This PR (5756) - mean (593ms)  : 577, 609
     .   : milestone, 593,
    master - mean (592ms)  : 578, 606
     .   : milestone, 592,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5756) - mean (191ms)  : 188, 194
     .   : milestone, 191,
    master - mean (191ms)  : 188, 195
     .   : milestone, 191,

    section CallTarget+Inlining+NGEN
    This PR (5756) - mean (1,004ms)  : 976, 1032
     .   : milestone, 1004,
    master - mean (1,006ms)  : 976, 1036
     .   : milestone, 1006,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5756) - mean (276ms)  : 272, 281
     .   : milestone, 276,
    master - mean (277ms)  : 272, 281
     .   : milestone, 277,

    section CallTarget+Inlining+NGEN
    This PR (5756) - mean (823ms)  : 792, 855
     .   : milestone, 823,
    master - mean (817ms)  : 797, 836
     .   : milestone, 817,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5756) - mean (264ms)  : 260, 268
     .   : milestone, 264,
    master - mean (265ms)  : 261, 269
     .   : milestone, 265,

    section CallTarget+Inlining+NGEN
    This PR (5756) - mean (811ms)  : 786, 835
     .   : milestone, 811,
    master - mean (805ms)  : 778, 832
     .   : milestone, 805,

datadog-ddstaging · 2024-07-01T10:51:49Z

Datadog Report

Branch report: andrew/ci/skipped-in-ci
Commit report: 804306a
Test service: dd-trace-dotnet

✅ 0 Failed, 344426 Passed, 2029 Skipped, 23h 45m 16.33s Total Time
❄️ 3 New Flaky

New Flaky Tests (3)

TestApiSecurityScan - Datadog.Trace.Security.IntegrationTests.ApiSecurity.AspNetCore5ApiSecurityDisabled
TestApiSecurityScan - Datadog.Trace.Security.IntegrationTests.ApiSecurity.AspNetCore5ApiSecurityDisabled
TestApiSecurityScan - Datadog.Trace.Security.IntegrationTests.ApiSecurity.AspNetCore5ApiSecurityDisabled

andrewlock · 2024-07-01T11:44:35Z

Throughput/Crank Report:zap:

Throughput results for AspNetCoreSimpleController comparing the following branches/commits:

Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.

Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!

gantt
    title Throughput Linux x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5756) (9.269M)   : 0, 9268802
    master (11.842M)   : 0, 11842369
    benchmarks/2.9.0 (12.049M)   : 0, 12048905

    section Automatic
    This PR (5756) (8.006M)   : 0, 8005594
    master (7.999M)   : 0, 7998666
    benchmarks/2.9.0 (8.464M)   : 0, 8463997

    section Trace stats
    master (8.337M)   : 0, 8336641

    section Manual
    This PR (5756) (10.284M)   : 0, 10284392
    master (10.173M)   : 0, 10173108

    section Manual + Automatic
    This PR (5756) (7.537M)   : 0, 7537174
    master (7.350M)   : 0, 7349541

    section Version Conflict
    master (6.703M)   : 0, 6702940

gantt
    title Throughput Linux arm64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5756) (7.765M)   : 0, 7764742
    master (9.512M)   : 0, 9511621
    benchmarks/2.9.0 (9.590M)   : 0, 9590349

    section Automatic
    This PR (5756) (6.414M)   : 0, 6414452
    master (6.515M)   : 0, 6515399

    section Trace stats
    master (6.862M)   : 0, 6861752

    section Manual
    This PR (5756) (8.367M)   : 0, 8367362
    master (8.299M)   : 0, 8298527

    section Manual + Automatic
    This PR (5756) (6.160M)   : 0, 6160370
    master (6.262M)   : 0, 6262373

    section Version Conflict
    master (5.588M)   : 0, 5588112

gantt
    title Throughput Windows x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5756) (7.918M)   : 0, 7918133
    master (10.162M)   : 0, 10161696
    benchmarks/2.9.0 (10.465M)   : 0, 10465033

    section Automatic
    This PR (5756) (7.197M)   : 0, 7197101
    master (7.293M)   : 0, 7293498
    benchmarks/2.9.0 (7.635M)   : 0, 7635248

    section Trace stats
    master (7.497M)   : 0, 7496532

    section Manual
    This PR (5756) (9.018M)   : 0, 9018441
    master (9.243M)   : 0, 9243291

    section Manual + Automatic
    This PR (5756) (6.973M)   : 0, 6972936
    master (6.976M)   : 0, 6975655

    section Version Conflict
    master (6.267M)   : 0, 6266553

NachoEchevarria

Thanks!

andrewlock · 2024-07-02T12:03:20Z

Benchmarks Report for tracer 🐌

Benchmarks for #5756 compared to master:

1 benchmarks are faster, with geometric mean 1.115
1 benchmarks are slower, with geometric mean 1.120
All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	7.76μs	44.3ns	319ns	0.0148	0.00738	0	5.42 KB
master	`StartStopWithChild`	netcoreapp3.1	10.2μs	54.2ns	302ns	0.0202	0.0101	0	5.62 KB
master	`StartStopWithChild`	net472	16.5μs	68.8ns	266ns	1.02	0.304	0.0961	6.06 KB
#5756	`StartStopWithChild`	net6.0	7.61μs	27.9ns	100ns	0.0157	0.00786	0	5.43 KB
#5756	`StartStopWithChild`	netcoreapp3.1	10.4μs	55.7ns	295ns	0.0155	0.00516	0	5.62 KB
#5756	`StartStopWithChild`	net472	16.4μs	79.2ns	307ns	1.02	0.297	0.0991	6.06 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	482μs	284ns	1.1μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	632μs	140ns	504ns	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	815μs	304ns	1.18μs	0.406	3.3 KB
#5756	`WriteAndFlushEnrichedTraces`	net6.0	485μs	167ns	626ns	0	2.7 KB
#5756	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	645μs	365ns	1.42μs	0	2.7 KB
#5756	`WriteAndFlushEnrichedTraces`	net472	833μs	75.8ns	284ns	0.417	3.3 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net6.0	174μs	185ns	718ns	0.259	18.45 KB
master	`SendRequest`	netcoreapp3.1	192μs	319ns	1.24μs	0.19	20.61 KB
master	`SendRequest`	net472	0ns	0ns	0ns	0	0 b
#5756	`SendRequest`	net6.0	170μs	236ns	915ns	0.169	18.45 KB
#5756	`SendRequest`	netcoreapp3.1	191μs	295ns	1.14μs	0.191	20.61 KB
#5756	`SendRequest`	net472	0.00117ns	0.000428ns	0.00166ns	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	568μs	929ns	3.6μs	0.558	0	0	41.66 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	650μs	1.12μs	4.34μs	0.324	0	0	41.71 KB
master	`WriteAndFlushEnrichedTraces`	net472	849μs	3.1μs	12μs	8.33	2.5	0.417	53.24 KB
#5756	`WriteAndFlushEnrichedTraces`	net6.0	580μs	1.49μs	5.78μs	0.553	0	0	41.84 KB
#5756	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	654μs	344ns	1.19μs	0.326	0	0	41.9 KB
#5756	`WriteAndFlushEnrichedTraces`	net472	836μs	2.92μs	11.3μs	8.45	2.53	0.422	53.25 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteNonQuery`	net6.0	1.17μs	0.881ns	3.41ns	0.0141	1.02 KB
master	`ExecuteNonQuery`	netcoreapp3.1	1.71μs	1.01ns	3.8ns	0.0137	1.02 KB
master	`ExecuteNonQuery`	net472	1.99μs	1.05ns	3.92ns	0.156	987 B
#5756	`ExecuteNonQuery`	net6.0	1.16μs	0.674ns	2.61ns	0.0147	1.02 KB
#5756	`ExecuteNonQuery`	netcoreapp3.1	1.69μs	1.42ns	5.51ns	0.0137	1.02 KB
#5756	`ExecuteNonQuery`	net472	1.92μs	2.06ns	7.98ns	0.156	987 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`CallElasticsearch`	net6.0	1.23μs	0.881ns	3.3ns	0.0136	976 B
master	`CallElasticsearch`	netcoreapp3.1	1.62μs	1.02ns	3.95ns	0.0131	976 B
master	`CallElasticsearch`	net472	2.63μs	2.32ns	8.68ns	0.157	995 B
master	`CallElasticsearchAsync`	net6.0	1.24μs	0.598ns	2.24ns	0.0129	952 B
master	`CallElasticsearchAsync`	netcoreapp3.1	1.7μs	0.614ns	2.21ns	0.0137	1.02 KB
master	`CallElasticsearchAsync`	net472	2.63μs	2.48ns	9.28ns	0.167	1.05 KB
#5756	`CallElasticsearch`	net6.0	1.14μs	0.407ns	1.52ns	0.0136	976 B
#5756	`CallElasticsearch`	netcoreapp3.1	1.57μs	2.63ns	10.2ns	0.013	976 B
#5756	`CallElasticsearch`	net472	2.57μs	1.9ns	7.37ns	0.158	995 B
#5756	`CallElasticsearchAsync`	net6.0	1.32μs	0.433ns	1.68ns	0.0132	952 B
#5756	`CallElasticsearchAsync`	netcoreapp3.1	1.65μs	0.778ns	3.01ns	0.0141	1.02 KB
#5756	`CallElasticsearchAsync`	net472	2.61μs	1.22ns	4.74ns	0.167	1.05 KB

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net6.0	1.34μs	0.495ns	1.78ns	0.0134	952 B
master	`ExecuteAsync`	netcoreapp3.1	1.58μs	0.59ns	2.21ns	0.0126	952 B
master	`ExecuteAsync`	net472	1.69μs	0.692ns	2.5ns	0.145	915 B
#5756	`ExecuteAsync`	net6.0	1.34μs	2.14ns	8.02ns	0.0134	952 B
#5756	`ExecuteAsync`	netcoreapp3.1	1.63μs	1.25ns	4.67ns	0.0123	952 B
#5756	`ExecuteAsync`	net472	1.71μs	0.591ns	2.29ns	0.145	915 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	4.01μs	1.42ns	5.48ns	0.0319	2.22 KB
master	`SendAsync`	netcoreapp3.1	4.93μs	1.24ns	4.63ns	0.0371	2.76 KB
master	`SendAsync`	net472	7.64μs	4.68ns	18.1ns	0.499	3.15 KB
#5756	`SendAsync`	net6.0	4.26μs	1.4ns	5.24ns	0.0298	2.22 KB
#5756	`SendAsync`	netcoreapp3.1	4.99μs	2.74ns	10.3ns	0.0361	2.76 KB
#5756	`SendAsync`	net472	7.66μs	7.62ns	29.5ns	0.496	3.15 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	1.56μs	0.522ns	1.95ns	0.0232	1.64 KB
master	`EnrichedLog`	netcoreapp3.1	2.18μs	2.63ns	10.2ns	0.0218	1.64 KB
master	`EnrichedLog`	net472	2.55μs	0.618ns	2.14ns	0.249	1.57 KB
#5756	`EnrichedLog`	net6.0	1.45μs	0.659ns	2.55ns	0.0226	1.64 KB
#5756	`EnrichedLog`	netcoreapp3.1	2.17μs	1.13ns	4.23ns	0.0227	1.64 KB
#5756	`EnrichedLog`	net472	2.64μs	0.706ns	2.55ns	0.249	1.57 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net6.0	115μs	174ns	674ns	0.0578	0	4.28 KB
master	`EnrichedLog`	netcoreapp3.1	120μs	88.6ns	331ns	0.0599	0	4.28 KB
master	`EnrichedLog`	net472	150μs	70.4ns	263ns	0.672	0.224	4.46 KB
#5756	`EnrichedLog`	net6.0	114μs	330ns	1.28μs	0	0	4.28 KB
#5756	`EnrichedLog`	netcoreapp3.1	120μs	312ns	1.21μs	0	0	4.28 KB
#5756	`EnrichedLog`	net472	147μs	130ns	503ns	0.661	0.22	4.46 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	3.02μs	0.731ns	2.83ns	0.0301	2.2 KB
master	`EnrichedLog`	netcoreapp3.1	4.21μs	2.14ns	8.29ns	0.0295	2.2 KB
master	`EnrichedLog`	net472	4.91μs	2.16ns	8.37ns	0.32	2.02 KB
#5756	`EnrichedLog`	net6.0	3μs	0.9ns	3.37ns	0.0301	2.2 KB
#5756	`EnrichedLog`	netcoreapp3.1	4.14μs	3.6ns	13.9ns	0.029	2.2 KB
#5756	`EnrichedLog`	net472	4.84μs	1.22ns	4.71ns	0.32	2.02 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendReceive`	net6.0	1.37μs	0.841ns	3.26ns	0.0158	1.14 KB
master	`SendReceive`	netcoreapp3.1	1.83μs	0.663ns	2.57ns	0.0148	1.14 KB
master	`SendReceive`	net472	2.06μs	1.4ns	5.24ns	0.183	1.16 KB
#5756	`SendReceive`	net6.0	1.46μs	0.729ns	2.82ns	0.0161	1.14 KB
#5756	`SendReceive`	netcoreapp3.1	1.69μs	0.689ns	2.48ns	0.0152	1.14 KB
#5756	`SendReceive`	net472	2μs	1ns	3.75ns	0.184	1.16 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.65μs	0.866ns	3.35ns	0.0226	1.6 KB
master	`EnrichedLog`	netcoreapp3.1	3.93μs	1.89ns	7.31ns	0.0217	1.65 KB
master	`EnrichedLog`	net472	4.44μs	2.57ns	9.96ns	0.323	2.04 KB
#5756	`EnrichedLog`	net6.0	2.8μs	0.771ns	2.89ns	0.0227	1.6 KB
#5756	`EnrichedLog`	netcoreapp3.1	3.93μs	4.17ns	16.2ns	0.0219	1.65 KB
#5756	`EnrichedLog`	net472	4.49μs	1.89ns	7.06ns	0.323	2.04 KB

Benchmarks.Trace.SpanBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #5756

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net6.0	1.120	485.92	544.19

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	446ns	0.183ns	0.709ns	0.00803	576 B
master	`StartFinishSpan`	netcoreapp3.1	537ns	0.267ns	0.998ns	0.00781	576 B
master	`StartFinishSpan`	net472	665ns	0.231ns	0.895ns	0.0916	578 B
master	`StartFinishScope`	net6.0	486ns	0.14ns	0.522ns	0.0098	696 B
master	`StartFinishScope`	netcoreapp3.1	723ns	0.768ns	2.77ns	0.00963	696 B
master	`StartFinishScope`	net472	921ns	0.73ns	2.83ns	0.105	658 B
#5756	`StartFinishSpan`	net6.0	480ns	0.193ns	0.695ns	0.00812	576 B
#5756	`StartFinishSpan`	netcoreapp3.1	553ns	0.226ns	0.875ns	0.00778	576 B
#5756	`StartFinishSpan`	net472	652ns	0.929ns	3.6ns	0.0918	578 B
#5756	`StartFinishScope`	net6.0	544ns	0.169ns	0.655ns	0.00982	696 B
#5756	`StartFinishScope`	netcoreapp3.1	735ns	0.187ns	0.674ns	0.00942	696 B
#5756	`StartFinishScope`	net472	880ns	0.414ns	1.49ns	0.104	658 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #5756

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin‑netcoreapp3.1	1.115	959.56	860.46

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	647ns	0.33ns	1.28ns	0.00963	696 B
master	`RunOnMethodBegin`	netcoreapp3.1	960ns	0.347ns	1.3ns	0.00941	696 B
master	`RunOnMethodBegin`	net472	1.07μs	0.656ns	2.54ns	0.105	658 B
#5756	`RunOnMethodBegin`	net6.0	694ns	0.301ns	1.17ns	0.00975	696 B
#5756	`RunOnMethodBegin`	netcoreapp3.1	861ns	0.401ns	1.45ns	0.00944	696 B
#5756	`RunOnMethodBegin`	net472	1.1μs	0.464ns	1.8ns	0.104	658 B

andrewlock · 2024-07-02T12:24:45Z

Benchmarks Report for appsec 🐌

Benchmarks for #5756 compared to master:

1 benchmarks are slower, with geometric mean 1.277
1 benchmarks have fewer allocations
1 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.Asm.AppSecBodyBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #5756

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorSimpleBody‑netcoreapp3.1	1.277	194.14	247.83

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`AllCycleSimpleBody`	net6.0	71.5μs	90.9ns	352ns	0.0713	0	6.01 KB
master	`AllCycleSimpleBody`	netcoreapp3.1	62.5μs	70.7ns	255ns	0.0929	0	6.95 KB
master	`AllCycleSimpleBody`	net472	47.8μs	58.8ns	228ns	1.3	0	8.34 KB
master	`AllCycleMoreComplexBody`	net6.0	80.3μs	93.2ns	361ns	0.117	0	9.51 KB
master	`AllCycleMoreComplexBody`	netcoreapp3.1	70.7μs	96.7ns	374ns	0.14	0	10.37 KB
master	`AllCycleMoreComplexBody`	net472	54.8μs	52.1ns	195ns	1.88	0.0273	11.85 KB
master	`ObjectExtractorSimpleBody`	net6.0	148ns	0.133ns	0.513ns	0.00397	0	280 B
master	`ObjectExtractorSimpleBody`	netcoreapp3.1	194ns	0.191ns	0.74ns	0.00371	0	272 B
master	`ObjectExtractorSimpleBody`	net472	170ns	0.095ns	0.368ns	0.0446	0	281 B
master	`ObjectExtractorMoreComplexBody`	net6.0	3.05μs	1.73ns	6.7ns	0.0535	0	3.78 KB
master	`ObjectExtractorMoreComplexBody`	netcoreapp3.1	4.05μs	1.82ns	6.8ns	0.0509	0	3.69 KB
master	`ObjectExtractorMoreComplexBody`	net472	4.16μs	2.59ns	9.68ns	0.602	0.00623	3.8 KB
#5756	`AllCycleSimpleBody`	net6.0	71.9μs	158ns	592ns	0.0718	0	6 KB
#5756	`AllCycleSimpleBody`	netcoreapp3.1	62.5μs	88.8ns	332ns	0.0941	0	6.95 KB
#5756	`AllCycleSimpleBody`	net472	47.8μs	68.2ns	264ns	1.31	0	8.34 KB
#5756	`AllCycleMoreComplexBody`	net6.0	78.1μs	160ns	621ns	0.117	0	9.51 KB
#5756	`AllCycleMoreComplexBody`	netcoreapp3.1	70.6μs	90.1ns	337ns	0.107	0	10.37 KB
#5756	`AllCycleMoreComplexBody`	net472	54.6μs	78.7ns	305ns	1.87	0.0271	11.85 KB
#5756	`ObjectExtractorSimpleBody`	net6.0	148ns	0.102ns	0.381ns	0.0039	0	280 B
#5756	`ObjectExtractorSimpleBody`	netcoreapp3.1	248ns	0.261ns	0.978ns	0.00371	0	272 B
#5756	`ObjectExtractorSimpleBody`	net472	169ns	0.198ns	0.74ns	0.0446	0	281 B
#5756	`ObjectExtractorMoreComplexBody`	net6.0	3.01μs	11.2ns	43.2ns	0.0523	0	3.78 KB
#5756	`ObjectExtractorMoreComplexBody`	netcoreapp3.1	4μs	1.5ns	5.61ns	0.0499	0	3.69 KB
#5756	`ObjectExtractorMoreComplexBody`	net472	3.84μs	2.34ns	8.76ns	0.601	0.0058	3.8 KB

Benchmarks.Trace.Asm.AppSecEncoderBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EncodeArgs`	net6.0	38.1μs	15.4ns	59.6ns	0.46	0	32.4 KB
master	`EncodeArgs`	netcoreapp3.1	55.3μs	25.5ns	98.9ns	0.414	0	32.4 KB
master	`EncodeArgs`	net472	67.2μs	52.6ns	197ns	5.16	0.0675	32.5 KB
master	`EncodeLegacyArgs`	net6.0	77.1μs	20.6ns	79.7ns	0	0	2.14 KB
master	`EncodeLegacyArgs`	netcoreapp3.1	106μs	79.1ns	306ns	0	0	2.14 KB
master	`EncodeLegacyArgs`	net472	156μs	68.7ns	266ns	0.311	0	2.15 KB
#5756	`EncodeArgs`	net6.0	38.5μs	17.6ns	68.3ns	0.46	0	32.4 KB
#5756	`EncodeArgs`	netcoreapp3.1	54.2μs	25ns	90.3ns	0.433	0	32.4 KB
#5756	`EncodeArgs`	net472	66.9μs	39.8ns	149ns	5.14	0.0668	32.5 KB
#5756	`EncodeLegacyArgs`	net6.0	80.5μs	37.6ns	146ns	0	0	2.14 KB
#5756	`EncodeLegacyArgs`	netcoreapp3.1	107μs	250ns	937ns	0	0	2.15 KB
#5756	`EncodeLegacyArgs`	net472	157μs	189ns	731ns	0.312	0	2.15 KB

Benchmarks.Trace.Asm.AppSecWafBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunWafRealisticBenchmark`	net6.0	185μs	42.3ns	164ns	0	2.42 KB
master	`RunWafRealisticBenchmark`	netcoreapp3.1	194μs	208ns	805ns	0	2.37 KB
master	`RunWafRealisticBenchmark`	net472	210μs	49.8ns	193ns	0.315	2.43 KB
master	`RunWafRealisticBenchmarkWithAttack`	net6.0	124μs	178ns	689ns	0	1.46 KB
master	`RunWafRealisticBenchmarkWithAttack`	netcoreapp3.1	129μs	77.5ns	290ns	0	1.45 KB
master	`RunWafRealisticBenchmarkWithAttack`	net472	141μs	29.6ns	107ns	0.212	1.48 KB
#5756	`RunWafRealisticBenchmark`	net6.0	184μs	61.5ns	238ns	0	2.42 KB
#5756	`RunWafRealisticBenchmark`	netcoreapp3.1	196μs	327ns	1.27μs	0	2.37 KB
#5756	`RunWafRealisticBenchmark`	net472	210μs	203ns	785ns	0.314	2.43 KB
#5756	`RunWafRealisticBenchmarkWithAttack`	net6.0	123μs	84ns	325ns	0	1.46 KB
#5756	`RunWafRealisticBenchmarkWithAttack`	netcoreapp3.1	128μs	28.5ns	98.8ns	0	1.45 KB
#5756	`RunWafRealisticBenchmarkWithAttack`	net472	140μs	80.6ns	312ns	0.21	1.48 KB

Benchmarks.Trace.Iast.StringAspectsBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #5756

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑netcoreapp3.1	254.01 KB	263.72 KB	9.71 KB	3.82%

Fewer allocations 🎉 in #5756

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑net6.0	263.3 KB	253.91 KB	-9.39 KB	-3.57%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Allocated
master	`StringConcatBenchmark`	net6.0	58.7μs	676ns	6.76μs	43.44 KB
master	`StringConcatBenchmark`	netcoreapp3.1	63.2μs	928ns	9.24μs	42.64 KB
master	`StringConcatBenchmark`	net472	39μs	104ns	388ns	59.26 KB
master	`StringConcatAspectBenchmark`	net6.0	273μs	5.17μs	51.2μs	263.3 KB
master	`StringConcatAspectBenchmark`	netcoreapp3.1	341μs	1.76μs	11.9μs	254.01 KB
master	`StringConcatAspectBenchmark`	net472	317μs	7.44μs	73.7μs	278.53 KB
#5756	`StringConcatBenchmark`	net6.0	59.5μs	781ns	7.77μs	43.44 KB
#5756	`StringConcatBenchmark`	netcoreapp3.1	52.9μs	168ns	582ns	42.64 KB
#5756	`StringConcatBenchmark`	net472	37.9μs	72.2ns	260ns	59.01 KB
#5756	`StringConcatAspectBenchmark`	net6.0	266μs	4.87μs	48μs	253.91 KB
#5756	`StringConcatAspectBenchmark`	netcoreapp3.1	334μs	1.09μs	3.77μs	263.72 KB
#5756	`StringConcatAspectBenchmark`	net472	314μs	8.04μs	78.4μs	278.53 KB

andrewlock added 2 commits July 1, 2024 10:39

Replace explicit skips with soft skips for non-flake

0ad27c5

Change default filters to all include (SkipInCI!=True)

804306a

andrewlock added the area:builds project files, build scripts, pipelines, versioning, releases, packages label Jul 1, 2024

andrewlock requested review from a team as code owners July 1, 2024 09:54

andrewlock mentioned this pull request Jul 1, 2024

Integration test for Oracle #5607

Merged

vandonr approved these changes Jul 1, 2024

View reviewed changes

kevingosse approved these changes Jul 1, 2024

View reviewed changes

NachoEchevarria approved these changes Jul 2, 2024

View reviewed changes

andrewlock merged commit 3b7c3f5 into master Jul 2, 2024
62 of 63 checks passed

andrewlock deleted the andrew/ci/skipped-in-ci branch July 2, 2024 15:08

github-actions bot added this to the vNext-v2 milestone Jul 2, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add attribute for skipping tests in CI without using `[Fact(Skip = "")]` #5756

Add attribute for skipping tests in CI without using `[Fact(Skip = "")]` #5756

andrewlock commented Jul 1, 2024

vandonr left a comment

andrewlock commented Jul 1, 2024

datadog-ddstaging bot commented Jul 1, 2024 •

edited

Loading

andrewlock commented Jul 1, 2024

NachoEchevarria left a comment

andrewlock commented Jul 2, 2024

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #5756

Raw results

Faster 🎉 in #5756

Raw results

andrewlock commented Jul 2, 2024

Slower ⚠️ in #5756

Raw results

Raw results

Raw results

More allocations ⚠️ in #5756

Fewer allocations 🎉 in #5756

Raw results

Add attribute for skipping tests in CI without using [Fact(Skip = "")] #5756

Add attribute for skipping tests in CI without using [Fact(Skip = "")] #5756

Conversation

andrewlock commented Jul 1, 2024

Summary of changes

Reason for change

Implementation details

Test coverage

Other details

vandonr left a comment

Choose a reason for hiding this comment

andrewlock commented Jul 1, 2024

Execution-Time Benchmarks Report ⏱️

datadog-ddstaging bot commented Jul 1, 2024 • edited Loading

Datadog Report

New Flaky Tests (3)

andrewlock commented Jul 1, 2024

Throughput/Crank Report:zap:

NachoEchevarria left a comment

Choose a reason for hiding this comment

andrewlock commented Jul 2, 2024

Benchmarks Report for tracer 🐌

Benchmark details

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #5756

Raw results

Faster 🎉 in #5756

Raw results

andrewlock commented Jul 2, 2024

Benchmarks Report for appsec 🐌

Benchmark details

Slower ⚠️ in #5756

Raw results

Raw results

Raw results

More allocations ⚠️ in #5756

Fewer allocations 🎉 in #5756

Raw results

Add attribute for skipping tests in CI without using `[Fact(Skip = "")]` #5756

Add attribute for skipping tests in CI without using `[Fact(Skip = "")]` #5756

datadog-ddstaging bot commented Jul 1, 2024 •

edited

Loading