Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Try to fix flake in hardcoded secrets test #4843

Merged
merged 3 commits into from
Nov 13, 2023
Merged

Try to fix flake in hardcoded secrets test #4843

merged 3 commits into from
Nov 13, 2023

Conversation

andrewlock
Copy link
Member

Summary of changes

  • Made HardcodedSecretsAnalyzer mostly non-static
  • Used a big timeout in the tests

Reason for change

We saw some flake in the unit tests due to timeouts.

Implementation details

Refactored the analyzer slightly so that things are stored in an instance instead of in a static. That means we can more easily pass in a timeout (and avoid shared state in general).

Used a fixture in the unit tests so we don't re-initialize the regexes for every test data

Test coverage

Covered by existing tests.

Other details

@andrewlock andrewlock requested review from a team as code owners November 9, 2023 13:57
@datadog-ddstaging
Copy link

datadog-ddstaging bot commented Nov 9, 2023 •
edited
Loading

Datadog Report

Branch report: andrew/ci/try-fix-hardcoded-secrets-flake
Commit report: 75d944c

❄️ dd-trace-dotnet: 0 Failed, 1 New Flaky, 295584 Passed, 973 Skipped, 46m 51.16s Wall Time

New Flaky Tests (1)

  • SubmitsTraces - Datadog.Trace.ClrProfiler.IntegrationTests.AWS.AwsKinesisTests

@andrewlock
Copy link
Member Author

andrewlock commented Nov 9, 2023 •
edited
Loading

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4843) - mean (72ms)  : 59, 85
     .   : milestone, 72,
    master - mean (70ms)  : 62, 79
     .   : milestone, 70,

    section CallTarget+Inlining+NGEN
    This PR (4843) - mean (1,003ms)  : 979, 1027
     .   : milestone, 1003,
    master - mean (993ms)  : 970, 1015
     .   : milestone, 993,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4843) - mean (107ms)  : 101, 112
     .   : milestone, 107,
    master - mean (106ms)  : 102, 110
     .   : milestone, 106,

    section CallTarget+Inlining+NGEN
    This PR (4843) - mean (683ms)  : 660, 706
     .   : milestone, 683,
    master - mean (689ms)  : 673, 706
     .   : milestone, 689,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4843) - mean (89ms)  : 85, 93
     .   : milestone, 89,
    master - mean (89ms)  : 85, 93
     .   : milestone, 89,

    section CallTarget+Inlining+NGEN
    This PR (4843) - mean (654ms)  : 630, 679
     .   : milestone, 654,
    master - mean (654ms)  : 628, 681
     .   : milestone, 654,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4843) - mean (188ms)  : 185, 191
     .   : milestone, 188,
    master - mean (187ms)  : 184, 190
     .   : milestone, 187,

    section CallTarget+Inlining+NGEN
    This PR (4843) - mean (1,103ms)  : 1081, 1124
     .   : milestone, 1103,
    master - mean (1,097ms)  : 1078, 1116
     .   : milestone, 1097,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4843) - mean (272ms)  : 268, 275
     .   : milestone, 272,
    master - mean (272ms)  : 268, 277
     .   : milestone, 272,

    section CallTarget+Inlining+NGEN
    This PR (4843) - mean (1,053ms)  : 1025, 1082
     .   : milestone, 1053,
    master - mean (1,054ms)  : 1027, 1081
     .   : milestone, 1054,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4843) - mean (263ms)  : 259, 267
     .   : milestone, 263,
    master - mean (262ms)  : 259, 265
     .   : milestone, 262,

    section CallTarget+Inlining+NGEN
    This PR (4843) - mean (1,025ms)  : 1000, 1051
     .   : milestone, 1025,
    master - mean (1,029ms)  : 1002, 1057
     .   : milestone, 1029,
Loading

@andrewlock andrewlock force-pushed the andrew/ci/try-fix-hardcoded-secrets-flake branch from 11904c0 to dfd1868 Compare November 9, 2023 16:50
@andrewlock
Copy link
Member Author

andrewlock commented Nov 9, 2023 •
edited
Loading

Benchmarks Report 🐌

Benchmarks for #4843 compared to master:

  • 3 benchmarks are faster, with geometric mean 1.281
  • 3 benchmarks are slower, with geometric mean 1.166
  • All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartStopWithChild net6.0 8.12μs 41.9ns 197ns 0.0199 0.00796 0 7.29 KB
master StartStopWithChild netcoreapp3.1 10.1μs 49.8ns 211ns 0.0197 0.00983 0 7.39 KB
master StartStopWithChild net472 15.5μs 56.1ns 217ns 1.3 0.325 0.1 7.65 KB
#4843 StartStopWithChild net6.0 7.99μs 42.6ns 221ns 0.0206 0.00824 0 7.28 KB
#4843 StartStopWithChild netcoreapp3.1 9.84μs 53.5ns 293ns 0.0204 0.0102 0 7.38 KB
#4843 StartStopWithChild net472 15.5μs 64.9ns 251ns 1.29 0.33 0.0997 7.66 KB
Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 481μs 262ns 1.01μs 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 647μs 385ns 1.49μs 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces net472 806μs 513ns 1.99μs 0.403 0 0 3.3 KB
#4843 WriteAndFlushEnrichedTraces net6.0 479μs 440ns 1.71μs 0 0 0 2.7 KB
#4843 WriteAndFlushEnrichedTraces netcoreapp3.1 647μs 278ns 1.08μs 0 0 0 2.7 KB
#4843 WriteAndFlushEnrichedTraces net472 795μs 258ns 931ns 0.398 0 0 3.3 KB
Benchmarks.Trace.Asm.AppSecBodyBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #4843

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorSimpleBody‑net6.0 1.427 189.00 132.41
Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorSimpleBody‑netcoreapp3.1 1.239 237.11 191.39

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody net6.0 43.1μs 114ns 440ns 0.0212 0 0 2.02 KB
master AllCycleSimpleBody netcoreapp3.1 45.8μs 45.1ns 163ns 0.0228 0 0 2 KB
master AllCycleSimpleBody net472 46.7μs 47.4ns 177ns 0.326 0 0 2.07 KB
master AllCycleMoreComplexBody net6.0 203μs 93.5ns 362ns 0.103 0 0 8.37 KB
master AllCycleMoreComplexBody netcoreapp3.1 210μs 153ns 594ns 0.105 0 0 8.26 KB
master AllCycleMoreComplexBody net472 214μs 51.8ns 187ns 1.28 0 0 8.43 KB
master ObjectExtractorSimpleBody net6.0 190ns 0.527ns 1.97ns 0.00396 0 0 280 B
master ObjectExtractorSimpleBody netcoreapp3.1 237ns 0.271ns 1.05ns 0.0037 0 0 272 B
master ObjectExtractorSimpleBody net472 156ns 0.0962ns 0.36ns 0.0446 0 0 281 B
master ObjectExtractorMoreComplexBody net6.0 2.82μs 1.13ns 4.24ns 0.0536 0 0 3.78 KB
master ObjectExtractorMoreComplexBody netcoreapp3.1 3.68μs 1.69ns 6.34ns 0.0497 0 0 3.69 KB
master ObjectExtractorMoreComplexBody net472 3.51μs 2.61ns 9.42ns 0.603 0.00527 0 3.8 KB
#4843 AllCycleSimpleBody net6.0 43μs 18.8ns 70.4ns 0.0217 0 0 2.02 KB
#4843 AllCycleSimpleBody netcoreapp3.1 45.7μs 53.8ns 208ns 0.0229 0 0 2 KB
#4843 AllCycleSimpleBody net472 47μs 37.1ns 144ns 0.324 0 0 2.07 KB
#4843 AllCycleMoreComplexBody net6.0 204μs 69.6ns 270ns 0.103 0 0 8.37 KB
#4843 AllCycleMoreComplexBody netcoreapp3.1 209μs 119ns 444ns 0.104 0 0 8.26 KB
#4843 AllCycleMoreComplexBody net472 213μs 136ns 525ns 1.28 0 0 8.43 KB
#4843 ObjectExtractorSimpleBody net6.0 133ns 0.086ns 0.322ns 0.00394 0 0 280 B
#4843 ObjectExtractorSimpleBody netcoreapp3.1 192ns 0.289ns 1.12ns 0.00361 0 0 272 B
#4843 ObjectExtractorSimpleBody net472 158ns 0.12ns 0.463ns 0.0446 0 0 281 B
#4843 ObjectExtractorMoreComplexBody net6.0 2.82μs 0.83ns 2.88ns 0.0533 0 0 3.78 KB
#4843 ObjectExtractorMoreComplexBody netcoreapp3.1 3.67μs 1.5ns 5.8ns 0.0513 0 0 3.69 KB
#4843 ObjectExtractorMoreComplexBody net472 3.6μs 2.76ns 10.7ns 0.603 0.00535 0 3.8 KB
Benchmarks.Trace.Asm.AppSecWafBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunWaf(args=NestedMap (10)) net6.0 13.1μs 2.44ns 8.81ns 0.13 0 0 9.42 KB
master RunWaf(args=NestedMap (10)) netcoreapp3.1 20.1μs 6.58ns 24.6ns 0.121 0 0 9.41 KB
master RunWaf(args=NestedMap (10)) net472 30μs 9.87ns 36.9ns 1.49 0.0149 0 9.47 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) net6.0 76.4μs 110ns 427ns 0.193 0 0 15.76 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) netcoreapp3.1 86.4μs 28.1ns 101ns 0.176 0 0 15.71 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) net472 99.4μs 184ns 711ns 2.5 0 0 16.03 KB
master RunWaf(args=NestedMap (100)) net6.0 24.3μs 19.3ns 74.7ns 0.268 0 0 19.66 KB
master RunWaf(args=NestedMap (100)) netcoreapp3.1 40.1μs 70.1ns 262ns 0.278 0 0 20.41 KB
master RunWaf(args=NestedMap (100)) net472 57.4μs 27.4ns 102ns 3.27 0.0573 0 20.62 KB
master RunWafWithAttack(args=Neste(...)tack) [23]) net6.0 92.5μs 40.6ns 157ns 0.37 0 0 26 KB
master RunWafWithAttack(args=Neste(...)tack) [23]) netcoreapp3.1 109μs 172ns 666ns 0.329 0 0 26.71 KB
master RunWafWithAttack(args=Neste(...)tack) [23]) net472 129μs 57.9ns 209ns 4.29 0.064 0 27.18 KB
master RunWaf(args=NestedMap (20)) net6.0 24.3μs 6.95ns 26.9ns 0.268 0 0 19.39 KB
master RunWaf(args=NestedMap (20)) netcoreapp3.1 39.5μs 10.3ns 40ns 0.256 0 0 19.83 KB
master RunWaf(args=NestedMap (20)) net472 56.8μs 24.8ns 92.7ns 3.18 0.0567 0 20.03 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) net6.0 91.9μs 140ns 543ns 0.321 0 0 25.74 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) netcoreapp3.1 111μs 191ns 688ns 0.331 0 0 26.14 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) net472 132μs 123ns 428ns 4.21 0.0658 0 26.59 KB
#4843 RunWaf(args=NestedMap (10)) net6.0 12.9μs 2.83ns 10.6ns 0.129 0 0 9.42 KB
#4843 RunWaf(args=NestedMap (10)) netcoreapp3.1 20.2μs 6.55ns 25.4ns 0.122 0 0 9.41 KB
#4843 RunWaf(args=NestedMap (10)) net472 30μs 13.4ns 51.9ns 1.5 0.015 0 9.47 KB
#4843 RunWafWithAttack(args=Neste(...)tack) [22]) net6.0 77.2μs 86ns 310ns 0.189 0 0 15.76 KB
#4843 RunWafWithAttack(args=Neste(...)tack) [22]) netcoreapp3.1 87.1μs 207ns 803ns 0.172 0 0 15.71 KB
#4843 RunWafWithAttack(args=Neste(...)tack) [22]) net472 97.6μs 26.2ns 94.3ns 2.53 0 0 16.03 KB
#4843 RunWaf(args=NestedMap (100)) net6.0 25μs 10.9ns 42.1ns 0.274 0 0 19.66 KB
#4843 RunWaf(args=NestedMap (100)) netcoreapp3.1 39.3μs 8.84ns 33.1ns 0.276 0 0 20.41 KB
#4843 RunWaf(args=NestedMap (100)) net472 56.9μs 22.6ns 84.6ns 3.27 0.0568 0 20.62 KB
#4843 RunWafWithAttack(args=Neste(...)tack) [23]) net6.0 90.4μs 43.1ns 161ns 0.359 0 0 26 KB
#4843 RunWafWithAttack(args=Neste(...)tack) [23]) netcoreapp3.1 110μs 239ns 829ns 0.328 0 0 26.71 KB
#4843 RunWafWithAttack(args=Neste(...)tack) [23]) net472 132μs 46.9ns 182ns 4.28 0.0658 0 27.18 KB
#4843 RunWaf(args=NestedMap (20)) net6.0 24.4μs 13ns 48.7ns 0.27 0 0 19.39 KB
#4843 RunWaf(args=NestedMap (20)) netcoreapp3.1 39μs 16ns 59.7ns 0.254 0 0 19.83 KB
#4843 RunWaf(args=NestedMap (20)) net472 56.8μs 34.8ns 130ns 3.18 0.0568 0 20.03 KB
#4843 RunWafWithAttack(args=Neste(...)tack) [22]) net6.0 91.6μs 92ns 356ns 0.364 0 0 25.74 KB
#4843 RunWafWithAttack(args=Neste(...)tack) [22]) netcoreapp3.1 111μs 135ns 506ns 0.334 0 0 26.14 KB
#4843 RunWafWithAttack(args=Neste(...)tack) [22]) net472 131μs 184ns 711ns 4.21 0.0648 0 26.59 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net6.0 166μs 115ns 444ns 0.25 0 0 18.04 KB
master SendRequest netcoreapp3.1 188μs 312ns 1.21μs 0.189 0 0 20.2 KB
master SendRequest net472 0.00021ns 9.31E‑05ns 0.00036ns 0 0 0 0 b
#4843 SendRequest net6.0 169μs 195ns 756ns 0.255 0 0 18.04 KB
#4843 SendRequest netcoreapp3.1 191μs 340ns 1.32μs 0.192 0 0 20.2 KB
#4843 SendRequest net472 9.96E‑05ns 9.96E‑05ns 0.000359ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 532μs 1.66μs 6.42μs 0.539 0 0 41.47 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 658μs 1.2μs 4.64μs 0.331 0 0 41.84 KB
master WriteAndFlushEnrichedTraces net472 840μs 3.24μs 12.5μs 8.28 2.48 0.414 53.26 KB
#4843 WriteAndFlushEnrichedTraces net6.0 533μs 322ns 1.21μs 0.541 0 0 41.5 KB
#4843 WriteAndFlushEnrichedTraces netcoreapp3.1 698μs 1.61μs 6.24μs 0.326 0 0 41.84 KB
#4843 WriteAndFlushEnrichedTraces net472 812μs 2.4μs 8.65μs 8.36 2.39 0.398 53.25 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net6.0 1μs 0.676ns 2.53ns 0.0105 0 0 768 B
master ExecuteNonQuery netcoreapp3.1 1.37μs 0.854ns 3.31ns 0.0101 0 0 768 B
master ExecuteNonQuery net472 1.62μs 0.648ns 2.43ns 0.115 0 0 730 B
#4843 ExecuteNonQuery net6.0 982ns 0.782ns 3.03ns 0.0109 0 0 768 B
#4843 ExecuteNonQuery netcoreapp3.1 1.31μs 5.41ns 21ns 0.0099 0 0 768 B
#4843 ExecuteNonQuery net472 1.65μs 0.776ns 3ns 0.116 0 0 730 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net6.0 1.16μs 0.687ns 2.57ns 0.0134 0 0 936 B
master CallElasticsearch netcoreapp3.1 1.35μs 0.386ns 1.5ns 0.0127 0 0 936 B
master CallElasticsearch net472 2.42μs 1.16ns 4.49ns 0.151 0 0 955 B
master CallElasticsearchAsync net6.0 1.24μs 0.616ns 2.39ns 0.013 0 0 912 B
master CallElasticsearchAsync netcoreapp3.1 1.43μs 0.997ns 3.73ns 0.0135 0 0 984 B
master CallElasticsearchAsync net472 2.34μs 0.455ns 1.76ns 0.16 0 0 1.01 KB
#4843 CallElasticsearch net6.0 1.23μs 0.728ns 2.62ns 0.013 0 0 936 B
#4843 CallElasticsearch netcoreapp3.1 1.35μs 0.474ns 1.84ns 0.0128 0 0 936 B
#4843 CallElasticsearch net472 2.35μs 0.411ns 1.54ns 0.152 0 0 955 B
#4843 CallElasticsearchAsync net6.0 1.25μs 0.57ns 2.05ns 0.0124 0 0 912 B
#4843 CallElasticsearchAsync netcoreapp3.1 1.4μs 1ns 3.88ns 0.0135 0 0 984 B
#4843 CallElasticsearchAsync net472 2.44μs 1.08ns 4.17ns 0.16 0 0 1.01 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net6.0 1.18μs 1.38ns 5.36ns 0.0129 0 0 912 B
master ExecuteAsync netcoreapp3.1 1.54μs 0.958ns 3.58ns 0.0124 0 0 912 B
master ExecuteAsync net472 1.64μs 3.95ns 15.3ns 0.139 0 0 875 B
#4843 ExecuteAsync net6.0 1.22μs 0.661ns 2.47ns 0.0127 0 0 912 B
#4843 ExecuteAsync netcoreapp3.1 1.47μs 0.498ns 1.86ns 0.0124 0 0 912 B
#4843 ExecuteAsync net472 1.7μs 0.488ns 1.76ns 0.138 0 0 875 B
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net6.0 3.86μs 2.44ns 9.12ns 0.0267 0 0 1.9 KB
master SendAsync netcoreapp3.1 4.55μs 1.79ns 6.93ns 0.0318 0 0 2.43 KB
master SendAsync net472 6.98μs 2.2ns 8.52ns 0.474 0 0 2.99 KB
#4843 SendAsync net6.0 3.78μs 1.97ns 7.37ns 0.0265 0 0 1.9 KB
#4843 SendAsync netcoreapp3.1 4.43μs 2.1ns 7.86ns 0.0333 0 0 2.43 KB
#4843 SendAsync net472 7.05μs 2.47ns 9.23ns 0.476 0 0 2.99 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 1.38μs 0.759ns 2.94ns 0.0218 0 0 1.57 KB
master EnrichedLog netcoreapp3.1 1.96μs 1.1ns 3.98ns 0.0206 0 0 1.57 KB
master EnrichedLog net472 2.35μs 3.77ns 14.1ns 0.236 0 0 1.49 KB
#4843 EnrichedLog net6.0 1.27μs 0.646ns 2.5ns 0.0223 0 0 1.57 KB
#4843 EnrichedLog netcoreapp3.1 1.9μs 1.06ns 4.12ns 0.0215 0 0 1.57 KB
#4843 EnrichedLog net472 2.19μs 2.74ns 10.6ns 0.236 0 0 1.49 KB
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 113μs 119ns 460ns 0 0 0 4.21 KB
master EnrichedLog netcoreapp3.1 119μs 143ns 535ns 0 0 0 4.21 KB
master EnrichedLog net472 149μs 72.2ns 270ns 0.67 0.223 0 4.38 KB
#4843 EnrichedLog net6.0 111μs 85.7ns 332ns 0.0557 0 0 4.21 KB
#4843 EnrichedLog netcoreapp3.1 116μs 106ns 412ns 0 0 0 4.21 KB
#4843 EnrichedLog net472 147μs 85.8ns 332ns 0.668 0.223 0 4.38 KB
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.83μs 0.911ns 3.53ns 0.0296 0 0 2.13 KB
master EnrichedLog netcoreapp3.1 3.76μs 0.911ns 3.29ns 0.0282 0 0 2.13 KB
master EnrichedLog net472 4.57μs 1.84ns 7.13ns 0.306 0 0 1.93 KB
#4843 EnrichedLog net6.0 3.1μs 1.07ns 4.16ns 0.0294 0 0 2.13 KB
#4843 EnrichedLog netcoreapp3.1 3.83μs 0.98ns 3.79ns 0.0287 0 0 2.13 KB
#4843 EnrichedLog net472 4.59μs 5.7ns 22.1ns 0.307 0 0 1.93 KB
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net6.0 1.34μs 0.821ns 3.18ns 0.0153 0 0 1.1 KB
master SendReceive netcoreapp3.1 1.57μs 1.13ns 4.23ns 0.0149 0 0 1.1 KB
master SendReceive net472 1.94μs 1.37ns 4.95ns 0.177 0 0 1.12 KB
#4843 SendReceive net6.0 1.29μs 0.747ns 2.89ns 0.0155 0 0 1.1 KB
#4843 SendReceive netcoreapp3.1 1.56μs 0.455ns 1.7ns 0.0148 0 0 1.1 KB
#4843 SendReceive net472 1.91μs 2.78ns 10.8ns 0.177 0 0 1.12 KB
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.65μs 0.727ns 2.72ns 0.0212 0 0 1.53 KB
master EnrichedLog netcoreapp3.1 3.53μs 3.43ns 13.3ns 0.0196 0 0 1.58 KB
master EnrichedLog net472 4.12μs 0.75ns 2.81ns 0.311 0 0 1.96 KB
#4843 EnrichedLog net6.0 2.69μs 0.916ns 3.55ns 0.0215 0 0 1.53 KB
#4843 EnrichedLog netcoreapp3.1 3.53μs 1.74ns 6.74ns 0.0211 0 0 1.58 KB
#4843 EnrichedLog net472 4.21μs 5.74ns 22.2ns 0.309 0 0 1.96 KB
Benchmarks.Trace.SpanBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #4843

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net472 1.191 622.58 741.48

Faster 🎉 in #4843

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net472 1.188 905.04 761.96

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net6.0 445ns 0.154ns 0.595ns 0.00741 0 0 536 B
master StartFinishSpan netcoreapp3.1 566ns 0.128ns 0.462ns 0.00738 0 0 536 B
master StartFinishSpan net472 622ns 0.294ns 1.14ns 0.0853 0 0 538 B
master StartFinishScope net6.0 460ns 0.144ns 0.558ns 0.00926 0 0 656 B
master StartFinishScope netcoreapp3.1 754ns 0.248ns 0.96ns 0.00865 0 0 656 B
master StartFinishScope net472 905ns 0.424ns 1.64ns 0.0978 0 0 618 B
#4843 StartFinishSpan net6.0 477ns 0.112ns 0.435ns 0.00756 0 0 536 B
#4843 StartFinishSpan netcoreapp3.1 564ns 0.229ns 0.886ns 0.00733 0 0 536 B
#4843 StartFinishSpan net472 742ns 0.15ns 0.562ns 0.0853 0 0 538 B
#4843 StartFinishScope net6.0 499ns 0.241ns 0.934ns 0.00909 0 0 656 B
#4843 StartFinishScope netcoreapp3.1 756ns 0.785ns 3.04ns 0.00892 0 0 656 B
#4843 StartFinishScope net472 761ns 0.682ns 2.64ns 0.0981 0 0 618 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #4843

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin‑netcoreapp3.1 1.182 728.97 861.79
Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin‑net6.0 1.125 524.61 590.22

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net6.0 525ns 0.247ns 0.955ns 0.00923 0 0 656 B
master RunOnMethodBegin netcoreapp3.1 729ns 0.229ns 0.886ns 0.00868 0 0 656 B
master RunOnMethodBegin net472 1.01μs 0.315ns 1.22ns 0.0982 0 0 618 B
#4843 RunOnMethodBegin net6.0 590ns 0.379ns 1.37ns 0.00901 0 0 656 B
#4843 RunOnMethodBegin netcoreapp3.1 862ns 0.457ns 1.71ns 0.00907 0 0 656 B
#4843 RunOnMethodBegin net472 987ns 1.09ns 4.22ns 0.0978 0 0 618 B

@andrewlock andrewlock force-pushed the andrew/ci/try-fix-hardcoded-secrets-flake branch from dfd1868 to 28e26dc Compare November 10, 2023 10:33
@andrewlock andrewlock force-pushed the andrew/ci/try-fix-hardcoded-secrets-flake branch from 28e26dc to 78b6af2 Compare November 10, 2023 14:17
@andrewlock andrewlock force-pushed the andrew/ci/try-fix-hardcoded-secrets-flake branch from 78b6af2 to 75d944c Compare November 10, 2023 16:21
@lucaspimentel lucaspimentel added area:tests unit tests, integration tests area:asm-iast labels Nov 10, 2023
@andrewlock
Copy link
Member Author

Throughput/Crank Report:zap:

Throughput results for AspNetCoreSimpleController comparing the following branches/commits:

Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.

Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!

gantt
    title Throughput Linux x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4843) (11.178M)   : 0, 11178386
    master (11.365M)   : 0, 11364660
    benchmarks/2.9.0 (11.650M)   : 0, 11650156

    section Automatic
    This PR (4843) (7.826M)   : 0, 7826083
    master (7.952M)   : 0, 7952033
    benchmarks/2.9.0 (8.463M)   : 0, 8463497

    section Trace stats
    This PR (4843) (8.044M)   : 0, 8043503
    master (8.125M)   : 0, 8124896

    section Manual
    This PR (4843) (9.868M)   : 0, 9867860
    master (9.996M)   : 0, 9996189

    section Manual + Automatic
    This PR (4843) (7.404M)   : 0, 7404282
    master (7.477M)   : 0, 7476818

    section Version Conflict
    This PR (4843) (6.793M)   : 0, 6793298
    master (6.908M)   : 0, 6907880
Loading 
gantt
    title Throughput Linux arm64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4843) (9.621M)   : 0, 9621174
    master (9.555M)   : 0, 9554556
    benchmarks/2.9.0 (9.454M)   : 0, 9454103

    section Automatic
    This PR (4843) (6.626M)   : 0, 6626224
    master (6.532M)   : 0, 6531809

    section Trace stats
    This PR (4843) (6.938M)   : 0, 6938496
    master (6.948M)   : 0, 6948235

    section Manual
    This PR (4843) (8.217M)   : 0, 8217202
    master (8.325M)   : 0, 8325246

    section Manual + Automatic
    This PR (4843) (6.372M)   : 0, 6371711
    master (6.277M)   : 0, 6277324

    section Version Conflict
    This PR (4843) (5.785M)   : 0, 5784566
    master (5.924M)   : 0, 5924344
Loading 
gantt
    title Throughput Windows x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4843) (10.234M)   : 0, 10233752
    master (10.211M)   : 0, 10211482
    benchmarks/2.9.0 (9.974M)   : 0, 9974322

    section Automatic
    This PR (4843) (7.258M)   : 0, 7257619
    master (7.507M)   : 0, 7507323
    benchmarks/2.9.0 (7.408M)   : 0, 7408367

    section Trace stats
    This PR (4843) (0.010M)   : crit ,0, 10381
    master (7.883M)   : 0, 7882755

    section Manual
    This PR (4843) (8.421M)   : crit ,0, 8420767
    master (9.456M)   : 0, 9456127

    section Manual + Automatic
    This PR (4843) (6.963M)   : 0, 6963223
    master (7.282M)   : 0, 7282409

    section Version Conflict
    This PR (4843) (6.390M)   : 0, 6390050
    master (6.615M)   : 0, 6615388
Loading 
gantt
    title Throughput Linux x64 (ASM) (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4843) (7.665M)   : 0, 7665043
    master (7.687M)   : 0, 7687290
    benchmarks/2.9.0 (7.940M)   : 0, 7940428

    section No attack
    This PR (4843) (2.195M)   : 0, 2195274
    master (2.179M)   : 0, 2179090
    benchmarks/2.9.0 (3.229M)   : 0, 3229263

    section Attack
    This PR (4843) (1.736M)   : 0, 1736076
    master (1.706M)   : 0, 1706183
    benchmarks/2.9.0 (2.541M)   : 0, 2540724

    section Blocking
    This PR (4843) (3.682M)   : 0, 3681827
    master (3.530M)   : 0, 3530105

    section IAST default
    This PR (4843) (6.947M)   : 0, 6947373
    master (6.727M)   : 0, 6726784

    section IAST full
    This PR (4843) (6.151M)   : 0, 6150824
    master (6.129M)   : 0, 6128654

    section Base vuln
    This PR (4843) (0.993M)   : 0, 993352
    master (0.990M)   : 0, 990312

    section IAST vuln
    This PR (4843) (0.864M)   : 0, 864121
    master (0.867M)   : 0, 867410
Loading

@andrewlock andrewlock merged commit c965bf0 into master Nov 13, 2023
54 checks passed
@andrewlock andrewlock deleted the andrew/ci/try-fix-hardcoded-secrets-flake branch November 13, 2023 10:35
@github-actions github-actions bot added this to the vNext milestone Nov 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@daniel-romano-DD daniel-romano-DD daniel-romano-DD approved these changes

@lucaspimentel lucaspimentel lucaspimentel approved these changes

Assignees
No one assigned
Labels
area:asm-iast area:tests unit tests, integration tests
Projects
None yet
Milestone
2.42.0
Development

Successfully merging this pull request may close these issues.
3 participants
@andrewlock @lucaspimentel @daniel-romano-DD