Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Tests] Ignore UDS telemetry test on windows #4926

Merged
merged 1 commit into from
Nov 27, 2023
Merged

[Tests] Ignore UDS telemetry test on windows #4926

merged 1 commit into from
Nov 27, 2023

Conversation

pierotibou
Copy link
Collaborator

Summary of changes

Skip UDS test telemetry test on windows

Reason for change

UDS isn't supported by the trace agent so it's not worth the hassle to fix this flaky test (happened twice last week)

Implementation details

Just skipped the test on windows.

Test coverage

Other details

@pierotibou pierotibou requested a review from a team as a code owner November 24, 2023 20:25
@github-actions github-actions bot added the area:tests unit tests, integration tests label Nov 24, 2023
@datadog-ddstaging
Copy link

datadog-ddstaging bot commented Nov 24, 2023 •
edited
Loading

Datadog Report

Branch report: pierre/windows-uds
Commit report: 47bea98

✅ dd-trace-dotnet: 0 Failed, 0 New Flaky, 299350 Passed, 951 Skipped, 30m 57.45s Wall Time

@andrewlock
Copy link
Member

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4926) - mean (71ms)  : 61, 80
     .   : milestone, 71,
    master - mean (71ms)  : 62, 80
     .   : milestone, 71,

    section CallTarget+Inlining+NGEN
    This PR (4926) - mean (986ms)  : 959, 1013
     .   : milestone, 986,
    master - mean (984ms)  : 957, 1011
     .   : milestone, 984,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4926) - mean (106ms)  : 103, 110
     .   : milestone, 106,
    master - mean (105ms)  : 100, 109
     .   : milestone, 105,

    section CallTarget+Inlining+NGEN
    This PR (4926) - mean (685ms)  : 669, 701
     .   : milestone, 685,
    master - mean (684ms)  : 665, 703
     .   : milestone, 684,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4926) - mean (90ms)  : 87, 93
     .   : milestone, 90,
    master - mean (89ms)  : 84, 94
     .   : milestone, 89,

    section CallTarget+Inlining+NGEN
    This PR (4926) - mean (656ms)  : 630, 681
     .   : milestone, 656,
    master - mean (659ms)  : 629, 689
     .   : milestone, 659,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4926) - mean (188ms)  : 185, 192
     .   : milestone, 188,
    master - mean (188ms)  : 184, 192
     .   : milestone, 188,

    section CallTarget+Inlining+NGEN
    This PR (4926) - mean (1,133ms)  : 1112, 1153
     .   : milestone, 1133,
    master - mean (1,132ms)  : 1109, 1154
     .   : milestone, 1132,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4926) - mean (271ms)  : 267, 276
     .   : milestone, 271,
    master - mean (271ms)  : 267, 275
     .   : milestone, 271,

    section CallTarget+Inlining+NGEN
    This PR (4926) - mean (1,088ms)  : 1068, 1108
     .   : milestone, 1088,
    master - mean (1,085ms)  : 1061, 1110
     .   : milestone, 1085,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4926) - mean (262ms)  : 258, 265
     .   : milestone, 262,
    master - mean (261ms)  : 258, 265
     .   : milestone, 261,

    section CallTarget+Inlining+NGEN
    This PR (4926) - mean (1,061ms)  : 1033, 1090
     .   : milestone, 1061,
    master - mean (1,056ms)  : 1032, 1079
     .   : milestone, 1056,
Loading

@andrewlock
Copy link
Member

Throughput/Crank Report:zap:

Throughput results for AspNetCoreSimpleController comparing the following branches/commits:

Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.

Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!

gantt
    title Throughput Linux x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4926) (10.896M)   : 0, 10896085
    master (11.399M)   : 0, 11399143
    benchmarks/2.9.0 (11.363M)   : 0, 11362982

    section Automatic
    This PR (4926) (7.412M)   : 0, 7412407
    master (7.766M)   : 0, 7765795
    benchmarks/2.9.0 (8.259M)   : 0, 8258618

    section Trace stats
    This PR (4926) (7.877M)   : 0, 7877357
    master (8.042M)   : 0, 8042224

    section Manual
    This PR (4926) (9.570M)   : 0, 9569914
    master (9.522M)   : 0, 9522223

    section Manual + Automatic
    This PR (4926) (7.230M)   : 0, 7229643
    master (7.447M)   : 0, 7447383

    section Version Conflict
    This PR (4926) (6.489M)   : 0, 6489415
    master (6.775M)   : 0, 6775133
Loading 
gantt
    title Throughput Linux arm64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4926) (9.490M)   : 0, 9490283
    master (9.789M)   : 0, 9789391
    benchmarks/2.9.0 (9.602M)   : 0, 9602224

    section Automatic
    This PR (4926) (6.545M)   : 0, 6545273
    master (6.529M)   : 0, 6529231

    section Trace stats
    This PR (4926) (6.984M)   : 0, 6983679
    master (6.918M)   : 0, 6917556

    section Manual
    This PR (4926) (8.417M)   : 0, 8416896
    master (8.224M)   : 0, 8223914

    section Manual + Automatic
    This PR (4926) (6.259M)   : 0, 6259452
    master (6.223M)   : 0, 6222722

    section Version Conflict
    This PR (4926) (5.646M)   : 0, 5645943
    master (5.659M)   : 0, 5658699
Loading 
gantt
    title Throughput Windows x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4926) (10.342M)   : 0, 10341766
    master (9.896M)   : 0, 9895779
    benchmarks/2.9.0 (10.259M)   : 0, 10259311

    section Automatic
    This PR (4926) (7.210M)   : 0, 7210482
    master (7.153M)   : 0, 7153186
    benchmarks/2.9.0 (7.710M)   : 0, 7709529

    section Trace stats
    This PR (4926) (7.545M)   : 0, 7545143
    master (7.475M)   : 0, 7475105

    section Manual
    This PR (4926) (8.980M)   : 0, 8980246
    master (8.673M)   : 0, 8672626

    section Manual + Automatic
    This PR (4926) (6.954M)   : 0, 6954376
    master (6.808M)   : 0, 6807618

    section Version Conflict
    This PR (4926) (6.381M)   : 0, 6380941
    master (6.119M)   : 0, 6119052
Loading 
gantt
    title Throughput Linux x64 (ASM) (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    master (7.421M)   : 0, 7420648
    benchmarks/2.9.0 (7.830M)   : 0, 7830154

    section No attack
    master (1.988M)   : 0, 1987679
    benchmarks/2.9.0 (3.253M)   : 0, 3253296

    section Attack
    master (1.584M)   : 0, 1583698
    benchmarks/2.9.0 (2.541M)   : 0, 2540898

    section Blocking
    master (3.198M)   : 0, 3197539

    section IAST default
    master (6.469M)   : 0, 6468524

    section IAST full
    master (5.804M)   : 0, 5803735

    section Base vuln
    master (0.938M)   : 0, 937552

    section IAST vuln
    master (0.878M)   : 0, 877600
Loading

@andrewlock
Copy link
Member

Benchmarks Report 🐌

Benchmarks for #4926 compared to master:

  • 1 benchmarks are faster, with geometric mean 1.191
  • 4 benchmarks are slower, with geometric mean 1.189
  • 1 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartStopWithChild net6.0 9.1μs 50.2ns 318ns 0.0219 0.00876 0 7.48 KB
master StartStopWithChild netcoreapp3.1 11.3μs 63.3ns 410ns 0.0233 0.0117 0 7.58 KB
master StartStopWithChild net472 17.3μs 43ns 167ns 1.33 0.361 0.103 7.94 KB
#4926 StartStopWithChild net6.0 8.57μs 44.8ns 301ns 0.02 0.00802 0 7.48 KB
#4926 StartStopWithChild netcoreapp3.1 10.6μs 58.3ns 340ns 0.0327 0.0164 0 7.57 KB
#4926 StartStopWithChild net472 17.1μs 67.4ns 261ns 1.34 0.341 0.111 7.95 KB
Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 461μs 190ns 736ns 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 632μs 111ns 401ns 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces net472 776μs 179ns 692ns 0.388 0 0 3.3 KB
#4926 WriteAndFlushEnrichedTraces net6.0 472μs 94.1ns 364ns 0 0 0 2.7 KB
#4926 WriteAndFlushEnrichedTraces netcoreapp3.1 633μs 281ns 973ns 0 0 0 2.7 KB
#4926 WriteAndFlushEnrichedTraces net472 779μs 241ns 903ns 0.388 0 0 3.3 KB
Benchmarks.Trace.Asm.AppSecBodyBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody net6.0 43.3μs 27ns 104ns 0.0221 0 0 1.77 KB
master AllCycleSimpleBody netcoreapp3.1 46.6μs 61.1ns 236ns 0.0236 0 0 1.74 KB
master AllCycleSimpleBody net472 49.9μs 108ns 390ns 0.268 0 0 1.81 KB
master AllCycleMoreComplexBody net6.0 215μs 67.3ns 243ns 0.107 0 0 9.25 KB
master AllCycleMoreComplexBody netcoreapp3.1 226μs 109ns 394ns 0.112 0 0 9.14 KB
master AllCycleMoreComplexBody net472 239μs 350ns 1.35μs 1.43 0 0 9.32 KB
master ObjectExtractorSimpleBody net6.0 132ns 0.0946ns 0.366ns 0.00394 0 0 280 B
master ObjectExtractorSimpleBody netcoreapp3.1 209ns 0.314ns 1.22ns 0.00375 0 0 272 B
master ObjectExtractorSimpleBody net472 151ns 0.0724ns 0.28ns 0.0446 0 0 281 B
master ObjectExtractorMoreComplexBody net6.0 2.77μs 1.26ns 4.71ns 0.0527 0 0 3.78 KB
master ObjectExtractorMoreComplexBody netcoreapp3.1 3.7μs 1.25ns 4.69ns 0.05 0 0 3.69 KB
master ObjectExtractorMoreComplexBody net472 3.49μs 1.92ns 7.44ns 0.603 0.00524 0 3.8 KB
#4926 AllCycleSimpleBody net6.0 43.9μs 32.1ns 124ns 0.022 0 0 1.77 KB
#4926 AllCycleSimpleBody netcoreapp3.1 47.2μs 149ns 576ns 0.0231 0 0 1.74 KB
#4926 AllCycleSimpleBody net472 50.1μs 16.8ns 65.1ns 0.276 0 0 1.81 KB
#4926 AllCycleMoreComplexBody net6.0 212μs 78.5ns 304ns 0.106 0 0 9.25 KB
#4926 AllCycleMoreComplexBody netcoreapp3.1 224μs 196ns 856ns 0.111 0 0 9.14 KB
#4926 AllCycleMoreComplexBody net472 236μs 116ns 434ns 1.43 0 0 9.32 KB
#4926 ObjectExtractorSimpleBody net6.0 131ns 0.0489ns 0.176ns 0.00393 0 0 280 B
#4926 ObjectExtractorSimpleBody netcoreapp3.1 225ns 0.0564ns 0.211ns 0.00375 0 0 272 B
#4926 ObjectExtractorSimpleBody net472 154ns 0.0526ns 0.204ns 0.0446 0 0 281 B
#4926 ObjectExtractorMoreComplexBody net6.0 2.8μs 1.6ns 5.98ns 0.0528 0 0 3.78 KB
#4926 ObjectExtractorMoreComplexBody netcoreapp3.1 3.66μs 1.48ns 5.73ns 0.0494 0 0 3.69 KB
#4926 ObjectExtractorMoreComplexBody net472 3.51μs 1.25ns 4.85ns 0.603 0.00527 0 3.8 KB
Benchmarks.Trace.Asm.AppSecWafBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #4926

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafWithAttack(args: NestedMap (20, attack))‑net6.0 1.142 163,050.49 186,246.46

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunWaf(args=NestedMap (10)) net6.0 52.3μs 263ns 1.15μs 0.214 0 0 16.06 KB
master RunWaf(args=NestedMap (10)) netcoreapp3.1 67.3μs 352ns 1.65μs 0.208 0 0 16.06 KB
master RunWaf(args=NestedMap (10)) net472 95.1μs 63.1ns 244ns 2.55 0.0946 0 16.14 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) net6.0 119μs 370ns 1.43μs 0.292 0 0 22.41 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) netcoreapp3.1 150μs 1.61μs 16.1μs 0.28 0 0 22.36 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) net472 169μs 843ns 3.58μs 3.53 0.164 0 22.7 KB
master RunWaf(args=NestedMap (100)) net6.0 93.8μs 277ns 1.07μs 0.465 0 0 32.76 KB
master RunWaf(args=NestedMap (100)) netcoreapp3.1 135μs 168ns 651ns 0.401 0 0 33.33 KB
master RunWaf(args=NestedMap (100)) net472 185μs 831ns 3.22μs 5.31 0.366 0 33.67 KB
master RunWafWithAttack(args=Neste(...)tack) [23]) net6.0 172μs 549ns 2.13μs 0.49 0 0 39.1 KB
master RunWafWithAttack(args=Neste(...)tack) [23]) netcoreapp3.1 205μs 1.07μs 5.46μs 0.528 0 0 39.63 KB
master RunWafWithAttack(args=Neste(...)tack) [23]) net472 267μs 1.05μs 4.07μs 6.34 0.518 0 40.23 KB
master RunWaf(args=NestedMap (20)) net6.0 100μs 255ns 989ns 0.403 0 0 32.18 KB
master RunWaf(args=NestedMap (20)) netcoreapp3.1 131μs 666ns 3.12μs 0.405 0 0 32.3 KB
master RunWaf(args=NestedMap (20)) net472 188μs 883ns 3.42μs 5.14 0.367 0 32.63 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) net6.0 163μs 78.4ns 283ns 0.517 0 0 38.53 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) netcoreapp3.1 206μs 1.01μs 4.04μs 0.42 0 0 38.6 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) net472 262μs 184ns 713ns 6.15 0.523 0 39.2 KB
#4926 RunWaf(args=NestedMap (10)) net6.0 49.3μs 25.8ns 100ns 0.219 0 0 16.06 KB
#4926 RunWaf(args=NestedMap (10)) netcoreapp3.1 69μs 392ns 2.32μs 0.195 0 0 16.06 KB
#4926 RunWaf(args=NestedMap (10)) net472 93.5μs 26.1ns 101ns 2.52 0.0933 0 16.14 KB
#4926 RunWafWithAttack(args=Neste(...)tack) [22]) net6.0 118μs 572ns 2.36μs 0.3 0 0 22.41 KB
#4926 RunWafWithAttack(args=Neste(...)tack) [22]) netcoreapp3.1 137μs 521ns 2.02μs 0.269 0 0 22.36 KB
#4926 RunWafWithAttack(args=Neste(...)tack) [22]) net472 169μs 656ns 2.54μs 3.54 0.165 0 22.7 KB
#4926 RunWaf(args=NestedMap (100)) net6.0 102μs 572ns 3.92μs 0.444 0 0 32.76 KB
#4926 RunWaf(args=NestedMap (100)) netcoreapp3.1 134μs 696ns 3.19μs 0.411 0 0 33.33 KB
#4926 RunWaf(args=NestedMap (100)) net472 183μs 63.5ns 246ns 5.31 0.366 0 33.67 KB
#4926 RunWafWithAttack(args=Neste(...)tack) [23]) net6.0 165μs 44ns 165ns 0.521 0 0 39.1 KB
#4926 RunWafWithAttack(args=Neste(...)tack) [23]) netcoreapp3.1 201μs 477ns 1.78μs 0.526 0 0 39.63 KB
#4926 RunWafWithAttack(args=Neste(...)tack) [23]) net472 267μs 817ns 3.17μs 6.38 0.521 0 40.23 KB
#4926 RunWaf(args=NestedMap (20)) net6.0 99.4μs 427ns 1.65μs 0.419 0 0 32.18 KB
#4926 RunWaf(args=NestedMap (20)) netcoreapp3.1 133μs 335ns 1.3μs 0.37 0 0 32.3 KB
#4926 RunWaf(args=NestedMap (20)) net472 181μs 83.9ns 325ns 5.14 0.361 0 32.63 KB
#4926 RunWafWithAttack(args=Neste(...)tack) [22]) net6.0 184μs 973ns 5.15μs 0.51 0 0 38.53 KB
#4926 RunWafWithAttack(args=Neste(...)tack) [22]) netcoreapp3.1 206μs 502ns 1.94μs 0.497 0 0 38.6 KB
#4926 RunWafWithAttack(args=Neste(...)tack) [22]) net472 260μs 160ns 622ns 6.19 0.515 0 39.2 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net6.0 173μs 123ns 477ns 0.259 0 0 18.24 KB
master SendRequest netcoreapp3.1 195μs 361ns 1.4μs 0.193 0 0 20.4 KB
master SendRequest net472 0ns 0ns 0ns 0 0 0 0 b
#4926 SendRequest net6.0 174μs 148ns 553ns 0.174 0 0 18.24 KB
#4926 SendRequest netcoreapp3.1 193μs 251ns 974ns 0.192 0 0 20.4 KB
#4926 SendRequest net472 7.44E‑05ns 5.59E‑05ns 0.000209ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 558μs 1.49μs 5.75μs 0.558 0 0 41.41 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 658μs 852ns 3.19μs 0.329 0 0 41.93 KB
master WriteAndFlushEnrichedTraces net472 831μs 2.78μs 10.4μs 8.13 2.57 0.428 53.22 KB
#4926 WriteAndFlushEnrichedTraces net6.0 554μs 1.05μs 4.07μs 0.561 0 0 41.6 KB
#4926 WriteAndFlushEnrichedTraces netcoreapp3.1 678μs 1.19μs 4.62μs 0.338 0 0 41.87 KB
#4926 WriteAndFlushEnrichedTraces net472 857μs 3.33μs 12.9μs 8.17 2.45 0.408 53.23 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net6.0 1.1μs 0.842ns 3.26ns 0.0105 0 0 768 B
master ExecuteNonQuery netcoreapp3.1 1.41μs 1.18ns 4.58ns 0.0103 0 0 768 B
master ExecuteNonQuery net472 1.79μs 0.755ns 2.92ns 0.115 0 0 730 B
#4926 ExecuteNonQuery net6.0 1.11μs 0.369ns 1.38ns 0.0106 0 0 768 B
#4926 ExecuteNonQuery netcoreapp3.1 1.49μs 0.584ns 2.26ns 0.00974 0 0 768 B
#4926 ExecuteNonQuery net472 1.84μs 3.98ns 15.4ns 0.116 0 0 730 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net6.0 1.23μs 0.554ns 2.15ns 0.0129 0 0 936 B
master CallElasticsearch netcoreapp3.1 1.53μs 1.83ns 6.84ns 0.0125 0 0 936 B
master CallElasticsearch net472 2.55μs 1.35ns 5.23ns 0.151 0 0 955 B
master CallElasticsearchAsync net6.0 1.28μs 0.494ns 1.91ns 0.0127 0 0 912 B
master CallElasticsearchAsync netcoreapp3.1 1.6μs 0.868ns 3.13ns 0.013 0 0 984 B
master CallElasticsearchAsync net472 2.71μs 0.922ns 3.57ns 0.16 0 0 1.01 KB
#4926 CallElasticsearch net6.0 1.33μs 0.899ns 3.36ns 0.0132 0 0 936 B
#4926 CallElasticsearch netcoreapp3.1 1.53μs 2.06ns 7.7ns 0.0122 0 0 936 B
#4926 CallElasticsearch net472 2.47μs 0.638ns 2.3ns 0.151 0 0 955 B
#4926 CallElasticsearchAsync net6.0 1.3μs 0.45ns 1.68ns 0.0123 0 0 912 B
#4926 CallElasticsearchAsync netcoreapp3.1 1.61μs 2.16ns 8.35ns 0.0135 0 0 984 B
#4926 CallElasticsearchAsync net472 2.51μs 0.658ns 2.55ns 0.16 0 0 1.01 KB
Benchmarks.Trace.GraphQLBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #4926

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.GraphQLBenchmark.ExecuteAsync‑net6.0 1.126 1,311.91 1,477.33

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net6.0 1.31μs 0.492ns 1.78ns 0.0125 0 0 912 B
master ExecuteAsync netcoreapp3.1 1.61μs 1.81ns 6.99ns 0.012 0 0 912 B
master ExecuteAsync net472 1.78μs 1.02ns 3.96ns 0.138 0 0 875 B
#4926 ExecuteAsync net6.0 1.48μs 0.744ns 2.78ns 0.0127 0 0 912 B
#4926 ExecuteAsync netcoreapp3.1 1.72μs 0.726ns 2.71ns 0.012 0 0 912 B
#4926 ExecuteAsync net472 1.88μs 4.9ns 19ns 0.139 0 0 875 B
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net6.0 4.15μs 0.798ns 2.99ns 0.0291 0 0 2.1 KB
master SendAsync netcoreapp3.1 5.06μs 2.07ns 8.03ns 0.0357 0 0 2.63 KB
master SendAsync net472 7.65μs 1.66ns 6.23ns 0.523 0 0 3.31 KB
#4926 SendAsync net6.0 4.12μs 1.36ns 5.29ns 0.0289 0 0 2.1 KB
#4926 SendAsync netcoreapp3.1 4.84μs 1.39ns 5.4ns 0.0363 0 0 2.63 KB
#4926 SendAsync net472 7.91μs 52ns 482ns 0.524 0 0 3.31 KB
Benchmarks.Trace.Iast.StringAspectsBenchmark - Slower ⚠️ More allocations ⚠️

Slower ⚠️ in #4926

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark(parameters: System.Collections.Generic.List`1[System.String])‑net472 1.313 48,600.00 63,800.00

More allocations ⚠️ in #4926

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark(parameters: System.Collections.Generic.List`1[System.String])‑net472 58.5 KB 59.59 KB 1.1 KB 1.87%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StringConcatBenchmark(parameters=Syste(...)ring] [48]) net6.0 51.3μs 240ns 898ns 0 0 0 43.44 KB
master StringConcatBenchmark(parameters=Syste(...)ring] [48]) netcoreapp3.1 53.3μs 164ns 636ns 0 0 0 42.64 KB
master StringConcatBenchmark(parameters=Syste(...)ring] [48]) net472 38μs 108ns 402ns 0 0 0 58.5 KB
master StringConcatAspectBenchmark(parameters=Syste(...)ring] [48]) net6.0 65.3μs 337ns 1.54μs 0 0 0 43.29 KB
master StringConcatAspectBenchmark(parameters=Syste(...)ring] [48]) netcoreapp3.1 104μs 4.01μs 39.9μs 0 0 0 42.64 KB
master StringConcatAspectBenchmark(parameters=Syste(...)ring] [48]) net472 48.5μs 194ns 671ns 0 0 0 57.34 KB
#4926 StringConcatBenchmark(parameters=Syste(...)ring] [48]) net6.0 54μs 208ns 749ns 0 0 0 43.44 KB
#4926 StringConcatBenchmark(parameters=Syste(...)ring] [48]) netcoreapp3.1 53.4μs 262ns 1.05μs 0 0 0 42.64 KB
#4926 StringConcatBenchmark(parameters=Syste(...)ring] [48]) net472 38.2μs 121ns 438ns 0 0 0 59.59 KB
#4926 StringConcatAspectBenchmark(parameters=Syste(...)ring] [48]) net6.0 97.4μs 2.89μs 28.9μs 0 0 0 43.29 KB
#4926 StringConcatAspectBenchmark(parameters=Syste(...)ring] [48]) netcoreapp3.1 104μs 2.82μs 27.8μs 0 0 0 42.64 KB
#4926 StringConcatAspectBenchmark(parameters=Syste(...)ring] [48]) net472 64.3μs 350ns 1.92μs 0 0 0 57.34 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 1.55μs 1.47ns 5.68ns 0.0217 0 0 1.57 KB
master EnrichedLog netcoreapp3.1 2.22μs 0.796ns 2.98ns 0.021 0 0 1.57 KB
master EnrichedLog net472 2.51μs 1.24ns 4.64ns 0.238 0 0 1.5 KB
#4926 EnrichedLog net6.0 1.55μs 0.639ns 2.39ns 0.022 0 0 1.57 KB
#4926 EnrichedLog netcoreapp3.1 2.24μs 0.824ns 2.97ns 0.0212 0 0 1.57 KB
#4926 EnrichedLog net472 2.61μs 2.21ns 8.57ns 0.238 0 0 1.5 KB
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 114μs 144ns 559ns 0.0566 0 0 4.21 KB
master EnrichedLog netcoreapp3.1 118μs 129ns 501ns 0 0 0 4.21 KB
master EnrichedLog net472 150μs 159ns 615ns 0.672 0.224 0 4.39 KB
#4926 EnrichedLog net6.0 112μs 122ns 457ns 0 0 0 4.21 KB
#4926 EnrichedLog netcoreapp3.1 118μs 214ns 830ns 0 0 0 4.21 KB
#4926 EnrichedLog net472 150μs 214ns 827ns 0.674 0.225 0 4.39 KB
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.78μs 0.757ns 2.73ns 0.0293 0 0 2.13 KB
master EnrichedLog netcoreapp3.1 4.19μs 5.59ns 21.6ns 0.0279 0 0 2.13 KB
master EnrichedLog net472 4.84μs 3.21ns 12ns 0.309 0 0 1.95 KB
#4926 EnrichedLog net6.0 2.9μs 0.937ns 3.63ns 0.0302 0 0 2.13 KB
#4926 EnrichedLog netcoreapp3.1 4.14μs 1.85ns 7.18ns 0.0291 0 0 2.13 KB
#4926 EnrichedLog net472 4.88μs 3.43ns 13.3ns 0.308 0 0 1.95 KB
Benchmarks.Trace.RedisBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #4926

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.RedisBenchmark.SendReceive‑net6.0 1.191 1,471.67 1,235.92

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net6.0 1.47μs 1.03ns 3.97ns 0.0155 0 0 1.1 KB
master SendReceive netcoreapp3.1 1.71μs 3.69ns 14.3ns 0.0145 0 0 1.1 KB
master SendReceive net472 2.13μs 1.51ns 5.84ns 0.177 0 0 1.12 KB
#4926 SendReceive net6.0 1.24μs 0.61ns 2.36ns 0.0155 0 0 1.1 KB
#4926 SendReceive netcoreapp3.1 1.72μs 0.896ns 3.35ns 0.0145 0 0 1.1 KB
#4926 SendReceive net472 2.19μs 3.96ns 15.3ns 0.177 0 0 1.12 KB
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.71μs 0.966ns 3.74ns 0.0217 0 0 1.53 KB
master EnrichedLog netcoreapp3.1 3.99μs 4.02ns 15.6ns 0.0198 0 0 1.58 KB
master EnrichedLog net472 4.34μs 1.7ns 6.59ns 0.312 0 0 1.97 KB
#4926 EnrichedLog net6.0 2.67μs 0.886ns 3.32ns 0.0213 0 0 1.53 KB
#4926 EnrichedLog netcoreapp3.1 3.95μs 1.64ns 6.37ns 0.0199 0 0 1.58 KB
#4926 EnrichedLog net472 4.23μs 0.902ns 3.25ns 0.312 0 0 1.97 KB
Benchmarks.Trace.SpanBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #4926

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net6.0 1.185 463.58 549.31

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net6.0 464ns 0.316ns 1.22ns 0.00741 0 0 536 B
master StartFinishSpan netcoreapp3.1 730ns 0.281ns 1.05ns 0.00729 0 0 536 B
master StartFinishSpan net472 739ns 0.245ns 0.95ns 0.0851 0 0 538 B
master StartFinishScope net6.0 604ns 0.222ns 0.86ns 0.00911 0 0 656 B
master StartFinishScope netcoreapp3.1 799ns 0.18ns 0.65ns 0.0089 0 0 656 B
master StartFinishScope net472 974ns 0.374ns 1.45ns 0.0977 0 0 618 B
#4926 StartFinishSpan net6.0 549ns 0.354ns 1.37ns 0.00742 0 0 536 B
#4926 StartFinishSpan netcoreapp3.1 768ns 1.96ns 7.59ns 0.00728 0 0 536 B
#4926 StartFinishSpan net472 719ns 0.358ns 1.39ns 0.0852 0 0 538 B
#4926 StartFinishScope net6.0 587ns 0.253ns 0.982ns 0.0091 0 0 656 B
#4926 StartFinishScope netcoreapp3.1 774ns 0.305ns 1.18ns 0.00903 0 0 656 B
#4926 StartFinishScope net472 977ns 0.971ns 3.76ns 0.0979 0 0 618 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net6.0 628ns 0.248ns 0.961ns 0.00945 0 0 656 B
master RunOnMethodBegin netcoreapp3.1 935ns 0.327ns 1.23ns 0.00889 0 0 656 B
master RunOnMethodBegin net472 1.17μs 0.704ns 2.72ns 0.0979 0 0 618 B
#4926 RunOnMethodBegin net6.0 620ns 0.264ns 1.02ns 0.00937 0 0 656 B
#4926 RunOnMethodBegin netcoreapp3.1 984ns 0.237ns 0.888ns 0.0089 0 0 656 B
#4926 RunOnMethodBegin net472 1.16μs 0.653ns 2.53ns 0.0976 0 0 618 B

andrewlock
andrewlock approved these changes Nov 27, 2023
View reviewed changes
Copy link
Member

@andrewlock andrewlock left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As loathed as I am to skip tests due to flake, it's hard to argue with you on this one😉

@pierotibou pierotibou merged commit 255ffcb into master Nov 27, 2023
54 checks passed
@pierotibou pierotibou deleted the pierre/windows-uds branch November 27, 2023 10:11
@github-actions github-actions bot added this to the vNext milestone Nov 27, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@andrewlock andrewlock andrewlock approved these changes

Assignees
No one assigned
Labels
area:tests unit tests, integration tests
Projects
None yet
Milestone
2.43.0
Development

Successfully merging this pull request may close these issues.
2 participants
@pierotibou @andrewlock