Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Linux/x64: 142 Regressions on 12/7/2022 10:10:38 AM #79607

Closed
performanceautofiler bot opened this issue Dec 13, 2022 · 10 comments
Closed

[Perf] Linux/x64: 142 Regressions on 12/7/2022 10:10:38 AM #79607

performanceautofiler bot opened this issue Dec 13, 2022 · 10 comments
Assignees
@vargaz
Labels
arch-x64 area-Codegen-Interpreter-mono os-linux Linux OS (any supported distro)
Milestone
8.0.0

Comments

@performanceautofiler
Copy link

Run Information

Architecture x64
OS ubuntu 18.04
Baseline b8584841ffed04ec8559629d0392eee9879f5dcb
Compare 59d56cc05b791bfb477ffc44cbef362287bd1054
Diff Diff

Regressions in System.Collections.TryGetValueFalse<Int32, Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SortedDictionary - Duration of single invocation 607.25 μs 692.09 μs 1.14 0.14 False
ImmutableSortedDictionary - Duration of single invocation 586.01 μs 648.61 μs 1.11 0.05 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.TryGetValueFalse&lt;Int32, Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.TryGetValueFalse<Int32, Int32>.SortedDictionary(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 692.0862336956523 > 653.6626083312499.
IsChangePoint: Marked as a change because one of 9/28/2022 6:40:13 PM, 10/22/2022 11:54:51 PM, 11/15/2022 8:48:03 PM, 11/22/2022 11:24:47 AM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -11.19507384735334 (T) = (0 -689448.823493863) / Math.Sqrt((484235957.5897148 / (31)) + (249520970.04535538 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.08809694114354276 = (633628.1239512374 - 689448.823493863) / 633628.1239512374 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.TryGetValueFalse&lt;Int32, Int32&gt;.ImmutableSortedDictionary(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 648.6063610491071 > 624.197248987611.
IsChangePoint: Marked as a change because one of 9/28/2022 3:03:11 PM, 10/22/2022 11:54:51 PM, 11/15/2022 8:48:03 PM, 11/22/2022 2:57:03 PM, 12/7/2022 10:10:38 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -9.201833885530117 (T) = (0 -641639.1238436721) / Math.Sqrt((252093372.5097592 / (32)) + (245576912.4583364 / (26))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (26) - 2, .025) and -0.06347841706644324 = (603340.0523666521 - 641639.1238436721) / 603340.0523666521 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline b8584841ffed04ec8559629d0392eee9879f5dcb
Compare 59d56cc05b791bfb477ffc44cbef362287bd1054
Diff Diff

Regressions in Benchstone.MDBenchI.MDGeneralArray

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Test - Duration of single invocation 938.11 ms 6.29 secs 6.71 0.15 False
Test2 - Duration of single invocation 934.69 ms 6.31 secs 6.75 0.12 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Benchstone.MDBenchI.MDGeneralArray*'

Payloads

Baseline
Compare

Histogram

Benchstone.MDBenchI.MDGeneralArray.Test

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 6.294134878285714 > 974.7867649853847.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -332.67252279848014 (T) = (0 -6324614195.630578) / Math.Sqrt((906468370819308.8 / (31)) + (6261031663079951 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -5.666072090617283 = (948776747.3341132 - 6324614195.630578) / 948776747.3341132 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### Benchstone.MDBenchI.MDGeneralArray.Test2

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 6.309850924928572 > 979.834198875.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -341.4035925796587 (T) = (0 -6285254191.324551) / Math.Sqrt((714591220814518.2 / (31)) + (5974629693366718 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -5.625040301109063 = (948711842.5939794 - 6285254191.324551) / 948711842.5939794 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline b8584841ffed04ec8559629d0392eee9879f5dcb
Compare 59d56cc05b791bfb477ffc44cbef362287bd1054
Diff Diff

Regressions in System.Collections.CreateAddAndClear<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
List - Duration of single invocation 13.03 μs 23.47 μs 1.80 0.07 False
HashSet - Duration of single invocation 74.27 μs 86.80 μs 1.17 0.05 False
Array - Duration of single invocation 5.16 μs 5.72 μs 1.11 0.04 False
SortedSet - Duration of single invocation 586.86 μs 640.27 μs 1.09 0.05 False
ImmutableArray - Duration of single invocation 261.26 μs 1.09 ms 4.17 0.02 False
ConcurrentBag - Duration of single invocation 72.90 μs 83.50 μs 1.15 0.06 False
IDictionary - Duration of single invocation 101.90 μs 115.97 μs 1.14 0.01 False
Queue - Duration of single invocation 25.43 μs 37.61 μs 1.48 0.11 False
SortedList - Duration of single invocation 452.70 μs 1.24 ms 2.73 0.00 False
Dictionary - Duration of single invocation 99.86 μs 111.87 μs 1.12 0.02 False
ICollection - Duration of single invocation 17.96 μs 29.35 μs 1.63 0.06 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CreateAddAndClear&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.CreateAddAndClear<Int32>.List(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 23.469186698893516 > 12.916729737048087.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -161.81748973999427 (T) = (0 -23729.039701293033) / Math.Sqrt((97733.2729470524 / (31)) + (45991.36346664534 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.9055328450654487 = (12452.70568950913 - 23729.039701293033) / 12452.70568950913 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndClear&lt;Int32&gt;.HashSet(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 86.8032005704365 > 78.426903206159.
IsChangePoint: Marked as a change because one of 9/23/2022 1:16:35 AM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -36.40434044961095 (T) = (0 -86958.13508971513) / Math.Sqrt((2217138.4295794037 / (31)) + (824857.8523305526 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.15440119547874265 = (75327.48184105323 - 86958.13508971513) / 75327.48184105323 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndClear&lt;Int32&gt;.Array(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 5.716856630642543 > 5.399100894675782.
IsChangePoint: Marked as a change because one of 11/15/2022 8:48:03 PM, 12/2/2022 5:21:30 AM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -18.48518076366622 (T) = (0 -5567.102932469006) / Math.Sqrt((6380.931601219815 / (31)) + (6821.039358783926 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.07653819458003623 = (5171.300898098432 - 5567.102932469006) / 5171.300898098432 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndClear&lt;Int32&gt;.SortedSet(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 640.2655003720239 > 616.7127253262997.
IsChangePoint: Marked as a change because one of 9/28/2022 6:40:13 PM, 10/22/2022 11:54:51 PM, 11/15/2022 8:48:03 PM, 11/22/2022 11:24:47 AM, 12/8/2022 3:49:24 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -12.01220410996348 (T) = (0 -643854.4397887815) / Math.Sqrt((398689911.64089644 / (36)) + (46558078.043755606 / (22))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (36) + (22) - 2, .025) and -0.07268521834314194 = (600226.8221643551 - 643854.4397887815) / 600226.8221643551 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndClear&lt;Int32&gt;.ImmutableArray(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.088982142857143 > 276.2522915484152.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -376.8357690560496 (T) = (0 -1090594.941926711) / Math.Sqrt((18660112.45729481 / (31)) + (113333022.77851015 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -3.1148981078260696 = (265035.71008296 - 1090594.941926711) / 265035.71008296 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndClear&lt;Int32&gt;.ConcurrentBag(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 83.5045912403475 > 76.60009291431061.
IsChangePoint: Marked as a change because one of 11/15/2022 8:48:03 PM, 12/2/2022 5:21:30 AM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -14.055342969139994 (T) = (0 -82297.57627283916) / Math.Sqrt((6139535.682714304 / (31)) + (2690300.2627694877 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.10275834182771805 = (74628.84038259795 - 82297.57627283916) / 74628.84038259795 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndClear&lt;Int32&gt;.IDictionary(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 115.9718737132353 > 107.65308224498571.
IsChangePoint: Marked as a change because one of 11/15/2022 8:48:03 PM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -25.98727133750973 (T) = (0 -115309.30339318319) / Math.Sqrt((3628105.640189915 / (31)) + (2778664.8968289616 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.11818768059609838 = (103121.60059902696 - 115309.30339318319) / 103121.60059902696 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndClear&lt;Int32&gt;.Queue(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 37.612352560240964 > 26.790697540098424.
IsChangePoint: Marked as a change because one of 11/15/2022 8:48:03 PM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -45.925900069924374 (T) = (0 -37014.780575339915) / Math.Sqrt((1434411.4025058036 / (31)) + (250737.15594331396 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.41333275150141474 = (26189.714018880757 - 37014.780575339915) / 26189.714018880757 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndClear&lt;Int32&gt;.SortedList(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.2351747310897436 > 481.12689615472993.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -301.61076696726565 (T) = (0 -1229816.142481173) / Math.Sqrt((55609320.484936915 / (31)) + (126180305.55516766 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -1.6573410010932557 = (462799.5210156373 - 1229816.142481173) / 462799.5210156373 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndClear&lt;Int32&gt;.Dictionary(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 111.87108105654761 > 104.90005048467356.
IsChangePoint: Marked as a change because one of 11/15/2022 10:02:59 PM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -39.77603222684615 (T) = (0 -112061.74309436271) / Math.Sqrt((1971209.5081188811 / (31)) + (634294.283447297 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.11699701161321148 = (100324.12077138746 - 112061.74309436271) / 100324.12077138746 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndClear&lt;Int32&gt;.ICollection(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 29.352911646110055 > 18.873768375699232.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -47.78328987979911 (T) = (0 -29802.996069191984) / Math.Sqrt((344930.27369587903 / (31)) + (1243141.1594205475 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.6216680859074272 = (18377.987658624545 - 29802.996069191984) / 18377.987658624545 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline b8584841ffed04ec8559629d0392eee9879f5dcb
Compare 59d56cc05b791bfb477ffc44cbef362287bd1054
Diff Diff

Regressions in System.Tests.Perf_Array

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ArrayRetrieve1D - Duration of single invocation 56.81 ms 92.47 ms 1.63 0.01 False
ArrayRetrieve3D - Duration of single invocation 79.96 ms 195.12 ms 2.44 0.01 False
ArrayAssign3D - Duration of single invocation 69.93 ms 171.79 ms 2.46 0.01 False
Clear - Duration of single invocation 802.50 ns 1.28 μs 1.60 0.04 False
ArrayCopy3D - Duration of single invocation 895.19 ns 2.51 μs 2.81 0.01 False
ArrayAssign1D - Duration of single invocation 54.61 ms 89.41 ms 1.64 0.01 False
ArrayAssign2D - Duration of single invocation 65.28 ms 132.76 ms 2.03 0.01 False
ArrayRetrieve2D - Duration of single invocation 67.16 ms 137.49 ms 2.05 0.01 False
ArrayCopy2D - Duration of single invocation 890.28 ns 2.52 μs 2.84 0.00 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Array*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Array.ArrayRetrieve1D

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 92.47110893333334 > 59.529360575000005.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -21.223216619456945 (T) = (0 -100087510.99728836) / Math.Sqrt((944226172326.4021 / (31)) + (107004198809894.9 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.7353657290510317 = (57675168.59516424 - 100087510.99728836) / 57675168.59516424 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Array.ArrayRetrieve3D

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 195.11992573333333 > 83.9455244775.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -54.40797583460208 (T) = (0 -202777833.6621286) / Math.Sqrt((1706486419546.1235 / (31)) + (135365602452713.36 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -1.5256843560719324 = (80286292.77234729 - 202777833.6621286) / 80286292.77234729 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Array.ArrayAssign3D

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 171.7920578 > 73.9266687490625.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -55.45285415461456 (T) = (0 -182884483.0767128) / Math.Sqrt((1110423131065.4656 / (31)) + (108905512668653.58 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -1.5750620889404228 = (71021387.74135944 - 182884483.0767128) / 71021387.74135944 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Array.Clear

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.2804935073141204 > 850.2766919782395.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -127.1820164510616 (T) = (0 -1281.975518770727) / Math.Sqrt((237.5497501364831 / (31)) + (146.7495100171736 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.5601731699878728 = (821.6879660741068 - 1281.975518770727) / 821.6879660741068 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Array.ArrayCopy3D

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.514441045422554 > 940.3953264838125.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -326.36903495578684 (T) = (0 -2511.4497234478554) / Math.Sqrt((92.51337904364681 / (31)) + (576.4641480733283 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -1.785979880695033 = (901.4601077525767 - 2511.4497234478554) / 901.4601077525767 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Array.ArrayAssign1D

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 89.40596875 > 57.93329311.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -21.44143410887711 (T) = (0 -97473520.43416937) / Math.Sqrt((760556966515.2164 / (31)) + (103192246062875.11 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.7587614009574852 = (55421685.04556896 - 97473520.43416937) / 55421685.04556896 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Array.ArrayAssign2D

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 132.7614871153846 > 67.79270777999999.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -40.95992139453779 (T) = (0 -142767105.53070477) / Math.Sqrt((921759610052.5035 / (31)) + (96164588354940.81 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -1.1915565594332844 = (65144157.43284443 - 142767105.53070477) / 65144157.43284443 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Array.ArrayRetrieve2D

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 137.49438343333333 > 70.714509425.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -38.38776010535931 (T) = (0 -144161378.61725006) / Math.Sqrt((573856769846.9851 / (31)) + (107603613011374.25 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -1.14050916337911 = (67349106.03684126 - 144161378.61725006) / 67349106.03684126 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Array.ArrayCopy2D

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.5241001749029435 > 937.8620140077794.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -324.6576433641876 (T) = (0 -2498.1992456941894) / Math.Sqrt((178.34028851076172 / (31)) + (501.1030098691688 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -1.7838377726305519 = (897.3939754160128 - 2498.1992456941894) / 897.3939754160128 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline b8584841ffed04ec8559629d0392eee9879f5dcb
Compare 59d56cc05b791bfb477ffc44cbef362287bd1054
Diff Diff

Regressions in System.Collections.CreateAddAndRemove<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SortedList - Duration of single invocation 2.67 ms 4.41 ms 1.65 0.02 False
Queue - Duration of single invocation 51.88 μs 64.29 μs 1.24 0.09 False
List - Duration of single invocation 167.39 μs 570.13 μs 3.41 0.05 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CreateAddAndRemove&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.CreateAddAndRemove<String>.SortedList(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 4.4123604302083335 > 2.8308888640625005.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -85.38138212497053 (T) = (0 -4277671.000971078) / Math.Sqrt((5664652178.633984 / (31)) + (4049769195.1352 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.5725245280355129 = (2720257.0927876006 - 4277671.000971078) / 2720257.0927876006 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndRemove&lt;String&gt;.Queue(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 64.28818438333332 > 54.06705439008348.
IsChangePoint: Marked as a change because one of 9/28/2022 9:30:44 PM, 11/15/2022 8:48:03 PM, 12/2/2022 5:21:30 AM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -24.59478322082287 (T) = (0 -63755.51149809263) / Math.Sqrt((4958665.781557593 / (31)) + (1047886.1980280007 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.20771189180473193 = (52790.33180903786 - 63755.51149809263) / 52790.33180903786 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndRemove&lt;String&gt;.List(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 570.1297382716049 > 175.73679156375115.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -260.4511859800968 (T) = (0 -560716.7040482993) / Math.Sqrt((16396587.887146804 / (31)) + (46053772.84641648 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -2.2718187228020907 = (171377.68059719505 - 560716.7040482993) / 171377.68059719505 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler
Copy link
Author

Run Information

Architecture x64
OS ubuntu 18.04
Baseline b8584841ffed04ec8559629d0392eee9879f5dcb
Compare 59d56cc05b791bfb477ffc44cbef362287bd1054
Diff Diff

Regressions in System.Collections.Tests.Perf_PriorityQueue<Int32, Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
HeapSort - Duration of single invocation 6.78 μs 8.50 μs 1.25 0.05 False
HeapSort - Duration of single invocation 2.21 ms 2.41 ms 1.09 0.04 False
HeapSort - Duration of single invocation 143.01 μs 153.20 μs 1.07 0.05 False
Dequeue_And_Enqueue - Duration of single invocation 5.09 ms 5.58 ms 1.10 0.03 False
K_Max_Elements - Duration of single invocation 21.16 μs 23.70 μs 1.12 0.05 False
K_Max_Elements - Duration of single invocation 4.37 μs 4.68 μs 1.07 0.03 False
Dequeue_And_Enqueue - Duration of single invocation 359.68 μs 389.52 μs 1.08 0.05 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Tests.Perf_PriorityQueue&lt;Int32, Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.Tests.Perf_PriorityQueue<Int32, Int32>.HeapSort(Size: 10)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 8.498380032260995 > 7.050516995996864.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -40.02973112078593 (T) = (0 -8413.64441307056) / Math.Sqrt((41770.70966720962 / (31)) + (7994.32107404939 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.2389730736072075 = (6790.821037437609 - 8413.64441307056) / 6790.821037437609 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Tests.Perf_PriorityQueue&lt;Int32, Int32&gt;.HeapSort(Size: 1000)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.407019285714286 > 2.3341874676510987.
IsChangePoint: Marked as a change because one of 9/28/2022 6:40:13 PM, 10/22/2022 11:54:51 PM, 11/15/2022 8:48:03 PM, 11/22/2022 11:24:47 AM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -9.349695425296822 (T) = (0 -2382277.5651759068) / Math.Sqrt((5383446421.668585 / (31)) + (1492332314.23674 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.06313125364692339 = (2240812.276944955 - 2382277.5651759068) / 2240812.276944955 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Tests.Perf_PriorityQueue&lt;Int32, Int32&gt;.HeapSort(Size: 100)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 153.20010596405228 > 148.33725513009543.
IsChangePoint: Marked as a change because one of 9/28/2022 6:40:13 PM, 10/22/2022 11:54:51 PM, 11/15/2022 8:48:03 PM, 11/22/2022 11:24:47 AM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -11.743569334852502 (T) = (0 -152782.29516868744) / Math.Sqrt((15686460.26013477 / (31)) + (5570872.216622533 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.06937480082469791 = (142870.67083576525 - 152782.29516868744) / 142870.67083576525 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Tests.Perf_PriorityQueue&lt;Int32, Int32&gt;.Dequeue_And_Enqueue(Size: 1000)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 5.582827906944444 > 5.322510740625.
IsChangePoint: Marked as a change because one of 9/28/2022 6:40:13 PM, 10/22/2022 11:54:51 PM, 11/15/2022 8:48:03 PM, 11/22/2022 11:24:47 AM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -10.68842368723691 (T) = (0 -5511619.371925875) / Math.Sqrt((24337979068.096397 / (31)) + (7190098317.565027 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.06710003469687946 = (5165044.693763416 - 5511619.371925875) / 5165044.693763416 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Tests.Perf_PriorityQueue&lt;Int32, Int32&gt;.K_Max_Elements(Size: 100)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 23.703692164510397 > 22.2252332952651.
IsChangePoint: Marked as a change because one of 9/28/2022 6:40:13 PM, 10/22/2022 11:54:51 PM, 11/15/2022 8:48:03 PM, 11/22/2022 11:24:47 AM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -8.622541366488877 (T) = (0 -22681.03342201256) / Math.Sqrt((427492.6922250421 / (31)) + (180736.0577046241 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.05754097170712214 = (21446.954802517 - 22681.03342201256) / 21446.954802517 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Tests.Perf_PriorityQueue&lt;Int32, Int32&gt;.K_Max_Elements(Size: 10)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 4.678673584540649 > 4.593034758109581.
IsChangePoint: Marked as a change because one of 9/28/2022 6:40:13 PM, 10/22/2022 11:54:51 PM, 11/15/2022 8:48:03 PM, 11/22/2022 2:57:03 PM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -9.15083727223579 (T) = (0 -4677.923187225349) / Math.Sqrt((12908.430193430262 / (31)) + (6787.814426815266 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.053242678143187935 = (4441.448570496862 - 4677.923187225349) / 4441.448570496862 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Tests.Perf_PriorityQueue&lt;Int32, Int32&gt;.Dequeue_And_Enqueue(Size: 100)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 389.51626390625 > 377.5074432938155.
IsChangePoint: Marked as a change because one of 9/28/2022 6:40:13 PM, 10/22/2022 11:54:51 PM, 11/15/2022 8:48:03 PM, 11/22/2022 11:24:47 AM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -9.383218992591805 (T) = (0 -384653.98090430174) / Math.Sqrt((135046851.27338502 / (31)) + (41547243.747822195 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.0629570366202018 = (361871.6163047895 - 384653.98090430174) / 361871.6163047895 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline b8584841ffed04ec8559629d0392eee9879f5dcb
Compare 59d56cc05b791bfb477ffc44cbef362287bd1054
Diff Diff

Regressions in System.Collections.Tests.Perf_BitArray

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
BitArrayIntArrayCtor - Duration of single invocation 376.78 ns 1.99 μs 5.29 0.05 False
BitArrayLengthCtor - Duration of single invocation 103.30 ns 112.12 ns 1.09 0.09 False
BitArrayCopyToIntArray - Duration of single invocation 158.33 ns 892.94 ns 5.64 0.17 False
BitArrayIntArrayCtor - Duration of single invocation 602.49 ns 2.22 μs 3.68 0.01 False
BitArrayLengthCtor - Duration of single invocation 100.31 ns 106.59 ns 1.06 0.07 False
BitArrayBitArrayCtor - Duration of single invocation 382.22 ns 1.98 μs 5.18 0.06 False
BitArrayBitArrayCtor - Duration of single invocation 380.94 ns 1.97 μs 5.18 0.04 False
BitArrayCopyToIntArray - Duration of single invocation 173.55 ns 904.90 ns 5.21 0.14 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Tests.Perf_BitArray*'

Payloads

Baseline
Compare

Histogram

System.Collections.Tests.Perf_BitArray.BitArrayIntArrayCtor(Size: 4)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.9914938324495601 > 394.51510599498215.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -358.1097840087689 (T) = (0 -1987.490628225535) / Math.Sqrt((97.15578524324471 / (31)) + (458.19873239356696 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -4.205563705884378 = (381.8012304755529 - 1987.490628225535) / 381.8012304755529 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Tests.Perf_BitArray.BitArrayLengthCtor(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 112.12075071046024 > 108.62242116753613.
IsChangePoint: Marked as a change because one of 11/15/2022 8:48:03 PM, 12/2/2022 10:04:21 AM, 12/10/2022 11:25:25 PM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -9.524186577512545 (T) = (0 -111.6781668911454) / Math.Sqrt((8.950973414592179 / (50)) + (2.954294422962015 / (8))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (50) + (8) - 2, .025) and -0.06740638237327459 = (104.62572524893454 - 111.6781668911454) / 104.62572524893454 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Tests.Perf_BitArray.BitArrayCopyToIntArray(Size: 4)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 892.9414673073426 > 168.22960782342446.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -238.86836510203946 (T) = (0 -890.5560362251141) / Math.Sqrt((46.17521753934575 / (31)) + (209.94055923713069 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -4.447706945126311 = (163.47355780982923 - 890.5560362251141) / 163.47355780982923 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Tests.Perf_BitArray.BitArrayIntArrayCtor(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.2185343954831174 > 632.9283219180966.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -277.5715776363328 (T) = (0 -2245.269395195628) / Math.Sqrt((162.1190082932573 / (31)) + (798.015854222081 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -2.6918649388011344 = (608.1667212681779 - 2245.269395195628) / 608.1667212681779 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Tests.Perf_BitArray.BitArrayLengthCtor(Size: 4)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 106.59267991270404 > 104.91193065535386.
IsChangePoint: Marked as a change because one of 11/15/2022 8:48:03 PM, 12/2/2022 5:21:30 AM, 12/10/2022 9:07:41 PM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -8.686934210117224 (T) = (0 -108.618697040457) / Math.Sqrt((4.483770737432742 / (49)) + (8.729440747986995 / (9))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (49) + (9) - 2, .025) and -0.0897956799254891 = (99.66886366064821 - 108.618697040457) / 99.66886366064821 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Tests.Perf_BitArray.BitArrayBitArrayCtor(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.979122975153007 > 407.1281672952314.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -488.8485587546002 (T) = (0 -1993.30969049026) / Math.Sqrt((90.25933766141203 / (31)) + (210.5884047786353 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -4.066743237963386 = (393.41044076500026 - 1993.30969049026) / 393.41044076500026 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Tests.Perf_BitArray.BitArrayBitArrayCtor(Size: 4)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.974990919226203 > 396.7857660503457.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -399.6891728398932 (T) = (0 -1990.2221546849814) / Math.Sqrt((117.17944503889251 / (31)) + (332.55712116679445 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -4.147602244528066 = (386.63091282948295 - 1990.2221546849814) / 386.63091282948295 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Tests.Perf_BitArray.BitArrayCopyToIntArray(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 904.8986120125081 > 183.0088647182973.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -235.74119781396163 (T) = (0 -902.5169888713667) / Math.Sqrt((57.766180945764894 / (31)) + (204.55775740319592 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -4.063894691453181 = (178.2258605011338 - 902.5169888713667) / 178.2258605011338 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline b8584841ffed04ec8559629d0392eee9879f5dcb
Compare 59d56cc05b791bfb477ffc44cbef362287bd1054
Diff Diff

Regressions in System.Collections.CreateAddAndRemove<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Queue - Duration of single invocation 46.60 μs 58.19 μs 1.25 0.09 False
HashSet - Duration of single invocation 155.91 μs 173.85 μs 1.12 0.02 False
SortedSet - Duration of single invocation 1.20 ms 1.29 ms 1.08 0.02 False
SortedDictionary - Duration of single invocation 1.43 ms 1.54 ms 1.07 0.05 False
Dictionary - Duration of single invocation 196.74 μs 215.81 μs 1.10 0.01 False
List - Duration of single invocation 126.40 μs 514.73 μs 4.07 0.07 False
SortedList - Duration of single invocation 893.76 μs 2.42 ms 2.70 0.01 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CreateAddAndRemove&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.CreateAddAndRemove<Int32>.Queue(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 58.186442998760846 > 49.01973910517724.
IsChangePoint: Marked as a change because one of 9/28/2022 6:40:13 PM, 11/15/2022 8:48:03 PM, 12/2/2022 10:04:21 AM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -20.781822238806967 (T) = (0 -58673.99220662765) / Math.Sqrt((4594746.635220089 / (31)) + (2990462.7415588805 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.21987920971154912 = (48098.198362198185 - 58673.99220662765) / 48098.198362198185 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndRemove&lt;Int32&gt;.HashSet(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 173.84775699786326 > 163.54338235850142.
IsChangePoint: Marked as a change because one of 9/23/2022 1:16:35 AM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -27.097353963753733 (T) = (0 -171783.6232502376) / Math.Sqrt((7775211.034582737 / (31)) + (2546220.934085182 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.10213214052252728 = (155864.81596371374 - 171783.6232502376) / 155864.81596371374 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndRemove&lt;Int32&gt;.SortedSet(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.2924417178819447 > 1.251612494579327.
IsChangePoint: Marked as a change because one of 9/28/2022 3:03:11 PM, 10/22/2022 11:54:51 PM, 11/15/2022 8:48:03 PM, 11/22/2022 11:24:47 AM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -12.040042196741517 (T) = (0 -1299778.3830619385) / Math.Sqrt((1164773808.2939403 / (31)) + (370306910.8706607 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.07105245669479286 = (1213552.4968337973 - 1299778.3830619385) / 1213552.4968337973 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndRemove&lt;Int32&gt;.SortedDictionary(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.5353186357142856 > 1.4884446487532783.
IsChangePoint: Marked as a change because one of 9/28/2022 6:40:13 PM, 10/11/2022 6:23:40 AM, 10/14/2022 1:19:37 PM, 10/22/2022 11:54:51 PM, 11/15/2022 8:48:03 PM, 11/22/2022 11:24:47 AM, 12/8/2022 9:11:55 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -10.138176977596961 (T) = (0 -1533658.2027104343) / Math.Sqrt((1752343391.3308465 / (37)) + (514768499.551967 / (21))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (21) - 2, .025) and -0.05936942891978617 = (1447708.5715738174 - 1533658.2027104343) / 1447708.5715738174 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndRemove&lt;Int32&gt;.Dictionary(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 215.81481368545835 > 206.73526429427082.
IsChangePoint: Marked as a change because one of 10/22/2022 11:54:51 PM, 11/15/2022 8:48:03 PM, 12/2/2022 5:21:30 AM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -19.730715900311694 (T) = (0 -216491.73870161918) / Math.Sqrt((15462682.191123642 / (31)) + (8447338.942506809 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.0894535245514233 = (198715.90097499432 - 216491.73870161918) / 198715.90097499432 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndRemove&lt;Int32&gt;.List(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 514.7315754608294 > 133.013580036886.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -210.6743632339307 (T) = (0 -515405.0152623476) / Math.Sqrt((23562105.184684474 / (31)) + (70675666.71032299 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -3.0197692511176037 = (128217.56251781147 - 515405.0152623476) / 128217.56251781147 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndRemove&lt;Int32&gt;.SortedList(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.4159301556547623 > 938.3885774131945.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -304.96931848383696 (T) = (0 -2404243.821047069) / Math.Sqrt((328218323.84828347 / (31)) + (375845403.17050654 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -1.687868431926528 = (894479.7269425233 - 2404243.821047069) / 894479.7269425233 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

Run Information

Architecture x64
OS ubuntu 18.04
Baseline b8584841ffed04ec8559629d0392eee9879f5dcb
Compare 59d56cc05b791bfb477ffc44cbef362287bd1054
Diff Diff

Regressions in System.IO.Tests.Perf_RandomAccess

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ReadScatterAsync - Duration of single invocation 33.37 ms 37.56 ms 1.13 0.19 False
ReadScatter - Duration of single invocation 421.93 μs 519.83 μs 1.23 0.02 False
ReadScatterAsync - Duration of single invocation 1.08 ms 1.19 ms 1.10 0.06 False
ReadScatter - Duration of single invocation 425.93 μs 519.97 μs 1.22 0.03 False
ReadScatter - Duration of single invocation 17.08 ms 19.61 ms 1.15 0.09 False
ReadScatter - Duration of single invocation 17.08 ms 18.78 ms 1.10 0.10 False
ReadScatterAsync - Duration of single invocation 1.11 ms 1.23 ms 1.10 0.04 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.IO.Tests.Perf_RandomAccess*'

Payloads

Baseline
Compare

Histogram

System.IO.Tests.Perf_RandomAccess.ReadScatterAsync(fileSize: 104857600, buffersSize: 65536, options: Asynchronous)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 37.561941025641026 > 35.214948691071434.
IsChangePoint: Marked as a change because one of 10/7/2022 6:01:23 PM, 11/22/2022 10:19:34 PM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -8.281513386617165 (T) = (0 -36552572.17035826) / Math.Sqrt((475705043966.0095 / (31)) + (2904503653367.3066 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.08628730837219045 = (33649083.33977736 - 36552572.17035826) / 33649083.33977736 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.Perf_RandomAccess.ReadScatter(fileSize: 1048576, buffersSize: 16384, options: None)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 519.8261639367817 > 445.20839487499995.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -35.701284475622 (T) = (0 -505721.56960614916) / Math.Sqrt((62057052.79082959 / (31)) + (75121283.26502216 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.1826049980371963 = (427633.5466580218 - 505721.56960614916) / 427633.5466580218 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.Perf_RandomAccess.ReadScatterAsync(fileSize: 1048576, buffersSize: 16384, options: Asynchronous)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.1935113477564103 > 1.1881602264843754.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/16/2022 6:51:59 PM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -17.44053895641235 (T) = (0 -1217516.8842748841) / Math.Sqrt((677677318.9065652 / (31)) + (347492965.7798112 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.09220295215325358 = (1114735.0241771247 - 1217516.8842748841) / 1114735.0241771247 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.Perf_RandomAccess.ReadScatter(fileSize: 1048576, buffersSize: 16384, options: Asynchronous)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 519.9680163177339 > 449.6339231465257.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -36.90223277021003 (T) = (0 -505464.1173237466) / Math.Sqrt((43122639.3796029 / (31)) + (68504074.55607043 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.1691771411575264 = (432324.6662377605 - 505464.1173237466) / 432324.6662377605 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.Perf_RandomAccess.ReadScatter(fileSize: 104857600, buffersSize: 65536, options: None)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 19.609700857142858 > 17.948946575892858.
IsChangePoint: Marked as a change because one of 11/16/2022 6:51:59 PM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -15.892683739147321 (T) = (0 -19164865.51409848) / Math.Sqrt((184715006190.8636 / (31)) + (291021761631.36096 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.12017615784716294 = (17108796.129825626 - 19164865.51409848) / 17108796.129825626 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.Perf_RandomAccess.ReadScatter(fileSize: 104857600, buffersSize: 65536, options: Asynchronous)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 18.777911994505494 > 17.870663775.
IsChangePoint: Marked as a change because one of 11/15/2022 8:48:03 PM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -15.649974740772018 (T) = (0 -19148018.56159534) / Math.Sqrt((270452829610.44733 / (31)) + (197956471687.91986 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.11552852900002851 = (17164974.32724542 - 19148018.56159534) / 17164974.32724542 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.Perf_RandomAccess.ReadScatterAsync(fileSize: 1048576, buffersSize: 16384, options: None)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.2271829322916665 > 1.1665631002232144.
IsChangePoint: Marked as a change because one of 10/10/2022 3:25:05 PM, 11/16/2022 11:42:33 AM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -22.37892708195564 (T) = (0 -1218993.1376660662) / Math.Sqrt((322377095.71977466 / (31)) + (265940671.30137518 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.0900503589470456 = (1118290.6621337892 - 1218993.1376660662) / 1118290.6621337892 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline b8584841ffed04ec8559629d0392eee9879f5dcb
Compare 59d56cc05b791bfb477ffc44cbef362287bd1054
Diff Diff

Regressions in System.Collections.CopyTo<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
List - Duration of single invocation 1.59 μs 2.33 μs 1.46 0.02 False
Array - Duration of single invocation 1.73 μs 3.35 μs 1.94 0.04 False
ImmutableArray - Duration of single invocation 1.74 μs 3.37 μs 1.93 0.05 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CopyTo&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.CopyTo<String>.List(Size: 2048)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.3259828310983877 > 1.6687268090711942.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -194.59422702518899 (T) = (0 -2329.365221948704) / Math.Sqrt((111.00306752890982 / (31)) + (292.1618415615554 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.4641879429386744 = (1590.892230182957 - 2329.365221948704) / 1590.892230182957 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CopyTo&lt;String&gt;.Array(Size: 2048)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.347677914685113 > 1.8128731089896546.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -275.92864835432084 (T) = (0 -3344.850443517928) / Math.Sqrt((492.6960833490932 / (31)) + (491.1423767389176 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.9290458849583119 = (1733.9403223113134 - 3344.850443517928) / 1733.9403223113134 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CopyTo&lt;String&gt;.ImmutableArray(Size: 2048)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.36671705185105 > 1.8335071039815602.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -364.5867055192175 (T) = (0 -3364.973570518579) / Math.Sqrt((29.20859368062028 / (31)) + (505.6915729361286 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.9251117115397488 = (1747.9367822385723 - 3364.973570518579) / 1747.9367822385723 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline b8584841ffed04ec8559629d0392eee9879f5dcb
Compare 59d56cc05b791bfb477ffc44cbef362287bd1054
Diff Diff

Regressions in System.Formats.Cbor.Tests.Perf_CborWriter

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
WriteCoseKey - Duration of single invocation 14.25 μs 16.39 μs 1.15 0.02 False
WriteCoseKey - Duration of single invocation 16.34 μs 18.29 μs 1.12 0.02 False
WriteCoseKey - Duration of single invocation 15.49 μs 17.28 μs 1.12 0.01 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Formats.Cbor.Tests.Perf_CborWriter*'

Payloads

Baseline
Compare

Histogram

System.Formats.Cbor.Tests.Perf_CborWriter.WriteCoseKey(publicKey: ECDSA_P256)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.38741954044392 > 15.368393598103536.
IsChangePoint: Marked as a change because one of 10/11/2022 6:23:40 AM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -20.607815940756907 (T) = (0 -16292.81032763179) / Math.Sqrt((134995.27587469315 / (31)) + (43231.37364520637 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.10817202545004014 = (14702.419798961368 - 16292.81032763179) / 14702.419798961368 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Formats.Cbor.Tests.Perf_CborWriter.WriteCoseKey(publicKey: ECDSA_P521)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 18.29215008333333 > 17.236721675300707.
IsChangePoint: Marked as a change because one of 10/11/2022 6:23:40 AM, 11/15/2022 8:48:03 PM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -18.621234093055563 (T) = (0 -18217.07915373895) / Math.Sqrt((152262.92244781365 / (31)) + (52341.671343282746 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.09242181220607773 = (16675.865448851386 - 18217.07915373895) / 16675.865448851386 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Formats.Cbor.Tests.Perf_CborWriter.WriteCoseKey(publicKey: ECDSA_P384)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 17.28388535892461 > 16.444159998904055.
IsChangePoint: Marked as a change because one of 10/11/2022 6:23:40 AM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -15.49923571234262 (T) = (0 -17346.999538899192) / Math.Sqrt((230457.92605969842 / (31)) + (58240.31203478407 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.09589374440856796 = (15829.088930753065 - 17346.999538899192) / 15829.088930753065 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline b8584841ffed04ec8559629d0392eee9879f5dcb
Compare 59d56cc05b791bfb477ffc44cbef362287bd1054
Diff Diff

Regressions in System.Collections.ContainsKeyTrue<Int32, Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Dictionary - Duration of single invocation 95.49 μs 104.23 μs 1.09 0.06 False
ConcurrentDictionary - Duration of single invocation 111.91 μs 122.42 μs 1.09 0.10 False
ImmutableSortedDictionary - Duration of single invocation 548.47 μs 588.01 μs 1.07 0.06 False
SortedDictionary - Duration of single invocation 554.32 μs 617.20 μs 1.11 0.13 False
IDictionary - Duration of single invocation 99.53 μs 107.83 μs 1.08 0.07 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsKeyTrue&lt;Int32, Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.ContainsKeyTrue<Int32, Int32>.Dictionary(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 104.22598052226029 > 100.71481146040982.
IsChangePoint: Marked as a change because one of 10/22/2022 11:54:51 PM, 11/15/2022 8:48:03 PM, 12/2/2022 5:21:30 AM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -10.005231782013746 (T) = (0 -105139.86627694461) / Math.Sqrt((10453724.982876943 / (31)) + (4688950.4569337545 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.07298138892958579 = (97988.5274448543 - 105139.86627694461) / 97988.5274448543 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.ContainsKeyTrue&lt;Int32, Int32&gt;.ConcurrentDictionary(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 122.41617309160304 > 121.38412563322372.
IsChangePoint: Marked as a change because one of 9/28/2022 6:40:13 PM, 10/11/2022 6:23:40 AM, 10/22/2022 11:54:51 PM, 11/15/2022 5:29:43 PM, 12/1/2022 11:48:13 PM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -8.654313759572876 (T) = (0 -125095.94984616431) / Math.Sqrt((20772279.394509044 / (31)) + (7393174.839956235 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.07205573507488804 = (116687.91626531038 - 125095.94984616431) / 116687.91626531038 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.ContainsKeyTrue&lt;Int32, Int32&gt;.ImmutableSortedDictionary(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 588.0117728174604 > 557.3331109213361.
IsChangePoint: Marked as a change because one of 9/28/2022 6:40:13 PM, 10/22/2022 11:54:51 PM, 11/15/2022 8:48:03 PM, 11/22/2022 11:24:47 AM, 12/7/2022 12:36:41 PM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -10.982326198779191 (T) = (0 -578510.1272735586) / Math.Sqrt((176702377.59620965 / (33)) + (210391829.2878478 / (25))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (25) - 2, .025) and -0.07578446619632319 = (537756.5353021044 - 578510.1272735586) / 537756.5353021044 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.ContainsKeyTrue&lt;Int32, Int32&gt;.SortedDictionary(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 617.2036455000001 > 592.6753986458334.
IsChangePoint: Marked as a change because one of 9/28/2022 6:40:13 PM, 10/22/2022 11:54:51 PM, 11/15/2022 8:48:03 PM, 11/22/2022 11:24:47 AM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -9.54861268613396 (T) = (0 -616531.4715919157) / Math.Sqrt((481721582.56683666 / (31)) + (94728255.54119125 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.07249416609469152 = (574857.6459272614 - 616531.4715919157) / 574857.6459272614 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.ContainsKeyTrue&lt;Int32, Int32&gt;.IDictionary(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 107.83178510273972 > 104.64330109201492.
IsChangePoint: Marked as a change because one of 10/22/2022 11:54:51 PM, 11/15/2022 8:48:03 PM, 12/2/2022 5:21:30 AM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -7.541213968620719 (T) = (0 -107487.24891053891) / Math.Sqrt((11116144.864149151 / (31)) + (6446724.436302444 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.05733396594953905 = (101658.74962127974 - 107487.24891053891) / 101658.74962127974 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

Run Information

Architecture x64
OS ubuntu 18.04
Baseline b8584841ffed04ec8559629d0392eee9879f5dcb
Compare 59d56cc05b791bfb477ffc44cbef362287bd1054
Diff Diff

Regressions in System.Collections.ContainsKeyFalse<Int32, Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SortedDictionary - Duration of single invocation 623.77 μs 736.18 μs 1.18 0.10 False
ImmutableSortedDictionary - Duration of single invocation 599.59 μs 672.82 μs 1.12 0.08 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsKeyFalse&lt;Int32, Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.ContainsKeyFalse<Int32, Int32>.SortedDictionary(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 736.1809372874148 > 663.6022499379808.
IsChangePoint: Marked as a change because one of 9/28/2022 3:03:11 PM, 10/22/2022 11:54:51 PM, 11/15/2022 8:48:03 PM, 11/22/2022 11:24:47 AM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -11.137504511521286 (T) = (0 -700427.0319276733) / Math.Sqrt((492319579.92146224 / (31)) + (304318528.1122283 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.09034236596818601 = (642391.8337849034 - 700427.0319276733) / 642391.8337849034 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.ContainsKeyFalse&lt;Int32, Int32&gt;.ImmutableSortedDictionary(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 672.8189473244146 > 626.8469589843751.
IsChangePoint: Marked as a change because one of 9/28/2022 6:40:13 PM, 10/22/2022 11:54:51 PM, 11/15/2022 8:48:03 PM, 11/22/2022 11:24:47 AM, 12/7/2022 8:33:47 PM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -8.751021495197582 (T) = (0 -640356.8161675474) / Math.Sqrt((323661308.63584644 / (35)) + (184574844.07084256 / (23))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (35) + (23) - 2, .025) and -0.06021544910349346 = (603987.4411460671 - 640356.8161675474) / 603987.4411460671 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline b8584841ffed04ec8559629d0392eee9879f5dcb
Compare 59d56cc05b791bfb477ffc44cbef362287bd1054
Diff Diff

Regressions in System.Collections.AddGivenSize<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SortedList - Duration of single invocation 447.27 μs 1.20 ms 2.68 0.01 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.AddGivenSize&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.AddGivenSize<Int32>.SortedList(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.200751237056213 > 476.4518171680571.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -248.67572836995396 (T) = (0 -1201327.2714497691) / Math.Sqrt((67216964.86177213 / (31)) + (183478092.71851602 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -1.6298584436125543 = (456803.0170473904 - 1201327.2714497691) / 456803.0170473904 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline b8584841ffed04ec8559629d0392eee9879f5dcb
Compare 59d56cc05b791bfb477ffc44cbef362287bd1054
Diff Diff

Regressions in LinqBenchmarks

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Where00ForX - Duration of single invocation 3.45 secs 5.05 secs 1.46 0.04 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'LinqBenchmarks*'

Payloads

Baseline
Compare

Histogram

LinqBenchmarks.Where00ForX

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 5.046679027785714 > 3.6188127997673076.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -63.6381185457108 (T) = (0 -5076105710.503968) / Math.Sqrt((18124838594777496 / (31)) + (934545889184367.8 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.4534540249042136 = (3492443258.2850337 - 5076105710.503968) / 3492443258.2850337 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

Run Information

Architecture x64
OS ubuntu 18.04
Baseline b8584841ffed04ec8559629d0392eee9879f5dcb
Compare 59d56cc05b791bfb477ffc44cbef362287bd1054
Diff Diff

Regressions in System.Collections.ContainsTrue<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
LinkedList - Duration of single invocation 12.32 ms 13.78 ms 1.12 0.07 False
ImmutableSortedSet - Duration of single invocation 524.94 μs 573.55 μs 1.09 0.04 False
SortedSet - Duration of single invocation 438.03 μs 475.69 μs 1.09 0.08 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsTrue&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.ContainsTrue<Int32>.LinkedList(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.775249955208333 > 12.8997506903125.
IsChangePoint: Marked as a change because one of 9/28/2022 6:40:13 PM, 10/11/2022 6:23:40 AM, 10/14/2022 1:19:37 PM, 10/22/2022 11:54:51 PM, 11/15/2022 8:48:03 PM, 11/22/2022 11:24:47 AM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -13.160894835779123 (T) = (0 -13256747.66231494) / Math.Sqrt((107861414904.2313 / (31)) + (40255746063.99615 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.07525839152115522 = (12328894.865503702 - 13256747.66231494) / 12328894.865503702 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.ContainsTrue&lt;Int32&gt;.ImmutableSortedSet(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 573.5484229497356 > 542.3974342948718.
IsChangePoint: Marked as a change because one of 9/28/2022 6:40:13 PM, 10/22/2022 11:54:51 PM, 11/15/2022 8:48:03 PM, 11/22/2022 11:24:47 AM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -12.77345840030968 (T) = (0 -557022.0199180511) / Math.Sqrt((156718069.04118785 / (31)) + (83797163.19343129 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.07009334777071599 = (520535.9150007553 - 557022.0199180511) / 520535.9150007553 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.ContainsTrue&lt;Int32&gt;.SortedSet(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 475.68952601010096 > 459.50238900240385.
IsChangePoint: Marked as a change because one of 9/28/2022 6:40:13 PM, 10/22/2022 11:54:51 PM, 11/15/2022 8:48:03 PM, 11/22/2022 11:24:47 AM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -12.667324993328137 (T) = (0 -483535.32447102416) / Math.Sqrt((124337490.8033106 / (31)) + (197246933.7538651 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.09664406221900822 = (440922.7580119414 - 483535.32447102416) / 440922.7580119414 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline b8584841ffed04ec8559629d0392eee9879f5dcb
Compare 59d56cc05b791bfb477ffc44cbef362287bd1054
Diff Diff

Regressions in System.Collections.CtorFromCollection<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Stack - Duration of single invocation 737.24 ns 1.99 μs 2.70 0.02 False
List - Duration of single invocation 725.24 ns 2.00 μs 2.76 0.01 False
ConcurrentBag - Duration of single invocation 76.26 μs 87.16 μs 1.14 0.06 False
ImmutableArray - Duration of single invocation 668.92 ns 1.51 μs 2.26 0.01 False
Queue - Duration of single invocation 733.20 ns 1.98 μs 2.70 0.02 False
ImmutableList - Duration of single invocation 135.90 μs 261.73 μs 1.93 0.04 False
ImmutableSortedDictionary - Duration of single invocation 1.85 ms 2.00 ms 1.08 0.02 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CtorFromCollection&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.CtorFromCollection<String>.Stack(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.9872387363682475 > 765.3898838545406.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -247.8560262879721 (T) = (0 -1986.5999174957656) / Math.Sqrt((191.18408803443805 / (31)) + (524.9117336220891 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -1.712708507613364 = (732.3307727019932 - 1986.5999174957656) / 732.3307727019932 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CtorFromCollection&lt;String&gt;.List(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.998847041693219 > 757.8254541997369.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -271.91050278002496 (T) = (0 -1971.2540371440498) / Math.Sqrt((182.0939067299553 / (31)) + (412.8474671926545 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -1.736604335627624 = (720.3284784287073 - 1971.2540371440498) / 720.3284784287073 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CtorFromCollection&lt;String&gt;.ConcurrentBag(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 87.1639379968844 > 81.70190904133803.
IsChangePoint: Marked as a change because one of 9/28/2022 6:40:13 PM, 11/15/2022 8:48:03 PM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -16.653825877307852 (T) = (0 -86786.30823765857) / Math.Sqrt((3841344.4784132666 / (31)) + (2905794.6730282386 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.10172962308028467 = (78772.78274048398 - 86786.30823765857) / 78772.78274048398 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CtorFromCollection&lt;String&gt;.ImmutableArray(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.5115079449602307 > 710.0802928665438.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -186.5164865543875 (T) = (0 -1481.3237185719317) / Math.Sqrt((90.88398462437446 / (31)) + (420.2856217658879 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -1.1812013464910682 = (679.1320393025429 - 1481.3237185719317) / 679.1320393025429 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CtorFromCollection&lt;String&gt;.Queue(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.9803473454518057 > 771.3335595266809.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -330.52101944617306 (T) = (0 -1991.938289494434) / Math.Sqrt((126.73974370720046 / (31)) + (276.89580322273054 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -1.691262879352783 = (740.1500257654026 - 1991.938289494434) / 740.1500257654026 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CtorFromCollection&lt;String&gt;.ImmutableList(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 261.7269954978813 > 145.9542556949013.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -155.35561938637795 (T) = (0 -264331.8216909534) / Math.Sqrt((10939226.754137365 / (31)) + (7873747.68753549 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.8933412687214447 = (139611.29251117742 - 264331.8216909534) / 139611.29251117742 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CtorFromCollection&lt;String&gt;.ImmutableSortedDictionary(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.0011202380208335 > 1.9433488372916667.
IsChangePoint: Marked as a change because one of 10/11/2022 6:23:40 AM, 11/15/2022 8:48:03 PM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -16.384858735136007 (T) = (0 -1997210.4737520905) / Math.Sqrt((1179083862.9035354 / (31)) + (820572941.731165 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.07280334240256087 = (1861674.3580228828 - 1997210.4737520905) / 1861674.3580228828 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline b8584841ffed04ec8559629d0392eee9879f5dcb
Compare 59d56cc05b791bfb477ffc44cbef362287bd1054
Diff Diff

Regressions in System.Linq.Tests.Perf_Enumerable

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ToDictionary - Duration of single invocation 23.80 μs 32.87 μs 1.38 0.02 False
OrderByThenBy - Duration of single invocation 64.91 μs 69.36 μs 1.07 0.03 False
ToList - Duration of single invocation 7.89 μs 15.78 μs 2.00 0.03 False
AnyWithPredicate_LastElementMatches - Duration of single invocation 60.15 μs 66.57 μs 1.11 0.03 False
ToList - Duration of single invocation 382.77 ns 1.57 μs 4.10 0.03 False
Distinct - Duration of single invocation 22.06 μs 29.93 μs 1.36 0.05 False
ToArray - Duration of single invocation 341.30 ns 1.53 μs 4.47 0.04 False
Except - Duration of single invocation 38.14 μs 47.35 μs 1.24 0.03 False
SelectToArray - Duration of single invocation 9.25 μs 16.26 μs 1.76 0.04 False
TakeLastHalf - Duration of single invocation 17.10 μs 23.29 μs 1.36 0.03 False
SequenceEqual - Duration of single invocation 18.16 μs 19.65 μs 1.08 0.14 False
ToArray - Duration of single invocation 7.59 μs 14.81 μs 1.95 0.05 False
SelectToList - Duration of single invocation 9.88 μs 17.32 μs 1.75 0.08 False
Intersect - Duration of single invocation 43.58 μs 52.06 μs 1.19 0.03 False
FirstWithPredicate_LastElementMatches - Duration of single invocation 60.37 μs 67.34 μs 1.12 0.05 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Linq.Tests.Perf_Enumerable*'

Payloads

Baseline
Compare

Histogram

System.Linq.Tests.Perf_Enumerable.ToDictionary(input: IEnumerable)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 32.8744465900456 > 24.903829422782877.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -74.6211823476377 (T) = (0 -32618.73275087648) / Math.Sqrt((192818.73802100262 / (31)) + (198537.4949190472 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.36337094418926197 = (23925.060813345583 - 32618.73275087648) / 23925.060813345583 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.OrderByThenBy(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 69.36376161644344 > 68.12998919197625.
IsChangePoint: Marked as a change because one of 9/28/2022 6:40:13 PM, 10/11/2022 6:23:40 AM, 11/15/2022 8:48:03 PM, 11/22/2022 11:24:47 AM, 12/2/2022 5:21:30 AM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -10.49798028294028 (T) = (0 -69574.55504061613) / Math.Sqrt((2588919.3078910494 / (31)) + (1341073.2850815405 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.05827440569588127 = (65743.39761610935 - 69574.55504061613) / 65743.39761610935 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.ToList(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.784582522344511 > 8.009778693583494.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -152.5070751352193 (T) = (0 -15976.014651275971) / Math.Sqrt((40076.79066340959 / (31)) + (42512.0241914733 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -1.0456723865975053 = (7809.664321591745 - 15976.014651275971) / 7809.664321591745 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.AnyWithPredicate_LastElementMatches(input: IOrderedEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 66.56716605616606 > 64.08263466398782.
IsChangePoint: Marked as a change because one of 9/28/2022 6:40:13 PM, 10/22/2022 11:54:51 PM, 11/15/2022 8:48:03 PM, 11/22/2022 11:24:47 AM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -12.920734323585247 (T) = (0 -65760.54961941809) / Math.Sqrt((3002956.1811200413 / (31)) + (291019.96133738256 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.06890724889164813 = (61521.28698500765 - 65760.54961941809) / 61521.28698500765 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.ToList(input: ICollection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.570486559378372 > 399.3550592984242.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -381.31339283667955 (T) = (0 -1564.3225702828408) / Math.Sqrt((104.91294445615709 / (31)) + (167.43681575434312 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -3.0764164275726475 = (383.7494520191442 - 1564.3225702828408) / 383.7494520191442 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.Distinct(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 29.93110407069738 > 23.461881264446433.
IsChangePoint: Marked as a change because one of 9/23/2022 1:16:35 AM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -47.974218423898556 (T) = (0 -29304.48287626955) / Math.Sqrt((337720.6300835146 / (31)) + (242278.6970985314 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.29997085519347766 = (22542.41528507814 - 29304.48287626955) / 22542.41528507814 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.ToArray(input: ICollection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.5251476119778562 > 354.2100221682117.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -383.6449300934785 (T) = (0 -1520.2010834732976) / Math.Sqrt((48.96678847347193 / (31)) + (213.8964733723423 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -3.5026405367651705 = (337.6243497699807 - 1520.2010834732976) / 337.6243497699807 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.Except(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 47.34592349941725 > 39.097488018966615.
IsChangePoint: Marked as a change because one of 9/23/2022 1:16:35 AM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -55.76874668952962 (T) = (0 -46645.86316905827) / Math.Sqrt((449468.39561550546 / (31)) + (320411.23036589765 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.24090026070893486 = (37590.34037305236 - 46645.86316905827) / 37590.34037305236 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.SelectToArray(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.264242719391365 > 9.675337026388585.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -143.54798710951658 (T) = (0 -16244.70432182537) / Math.Sqrt((50179.70161741367 / (31)) + (19223.473474251517 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.7440006489142147 = (9314.620571924126 - 16244.70432182537) / 9314.620571924126 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.TakeLastHalf(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 23.28511948845328 > 18.096738732435803.
IsChangePoint: Marked as a change because one of 11/15/2022 8:48:03 PM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -46.31866885749217 (T) = (0 -23591.968445270988) / Math.Sqrt((399219.23086676135 / (31)) + (109603.44540041451 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.3432095615993928 = (17563.877685012492 - 23591.968445270988) / 17563.877685012492 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.SequenceEqual(input1: IEnumerable, input2: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 19.645180854978353 > 19.325720286093507.
IsChangePoint: Marked as a change because one of 10/22/2022 11:54:51 PM, 11/15/2022 8:48:03 PM, 11/22/2022 11:24:47 AM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -9.923797627962971 (T) = (0 -20274.485221238647) / Math.Sqrt((398717.14245050296 / (31)) + (304891.62209161907 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.08233519210687414 = (18732.16852699054 - 20274.485221238647) / 18732.16852699054 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.ToArray(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 14.806591185591603 > 8.037984892771327.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -114.02887887105771 (T) = (0 -14812.191468624034) / Math.Sqrt((100194.7416130558 / (31)) + (15469.153368601626 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.904287502543624 = (7778.33780289944 - 14812.191468624034) / 7778.33780289944 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.SelectToList(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 17.31613121376655 > 10.07259810150186.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -129.6649922542151 (T) = (0 -17519.793710178143) / Math.Sqrt((67371.52124000604 / (31)) + (47375.13150045695 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.8651382311990891 = (9393.29504758194 - 17519.793710178143) / 9393.29504758194 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.Intersect(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 52.0621141826923 > 44.16643008643581.
IsChangePoint: Marked as a change because one of 9/23/2022 1:16:35 AM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -38.31162625305965 (T) = (0 -51704.1517161656) / Math.Sqrt((1383866.1294066731 / (31)) + (359229.8006794512 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.21708906684488125 = (42481.814293345655 - 51704.1517161656) / 42481.814293345655 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.FirstWithPredicate_LastElementMatches(input: IOrderedEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 67.34150973865877 > 63.53881734132753.
IsChangePoint: Marked as a change because one of 9/28/2022 6:40:13 PM, 10/17/2022 12:47:34 PM, 11/15/2022 8:48:03 PM, 11/22/2022 11:24:47 AM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -12.200664371117858 (T) = (0 -65987.63924007337) / Math.Sqrt((3286444.315163871 / (31)) + (509371.6240348758 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.06990572697280452 = (61676.124892591295 - 65987.63924007337) / 61676.124892591295 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline b8584841ffed04ec8559629d0392eee9879f5dcb
Compare 59d56cc05b791bfb477ffc44cbef362287bd1054
Diff Diff

Regressions in System.Collections.TryAddDefaultSize<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Dictionary - Duration of single invocation 156.80 μs 177.18 μs 1.13 0.01 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.TryAddDefaultSize&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.TryAddDefaultSize<String>.Dictionary(Count: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 177.18105744047622 > 168.1960553005337.
IsChangePoint: Marked as a change because one of 9/28/2022 6:40:13 PM, 11/15/2022 8:48:03 PM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -21.80369422593142 (T) = (0 -174010.85103754885) / Math.Sqrt((7453212.31258739 / (31)) + (2646002.5952775925 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.07862414629089234 = (161326.67865440142 - 174010.85103754885) / 161326.67865440142 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

Run Information

Architecture x64
OS ubuntu 18.04
Baseline b8584841ffed04ec8559629d0392eee9879f5dcb
Compare 59d56cc05b791bfb477ffc44cbef362287bd1054
Diff Diff

Regressions in System.Collections.ContainsFalse<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SortedSet - Duration of single invocation 482.35 μs 554.74 μs 1.15 0.10 False
LinkedList - Duration of single invocation 24.05 ms 26.05 ms 1.08 0.07 False
ImmutableSortedSet - Duration of single invocation 584.81 μs 657.90 μs 1.12 0.05 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsFalse&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.ContainsFalse<Int32>.SortedSet(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 554.7399172512755 > 511.1309991552734.
IsChangePoint: Marked as a change because one of 9/28/2022 6:40:13 PM, 10/22/2022 11:54:51 PM, 11/15/2022 8:48:03 PM, 11/22/2022 11:24:47 AM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -12.495316803132614 (T) = (0 -539589.7019980092) / Math.Sqrt((209204087.62664497 / (31)) + (120084092.74938276 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.08399299161468933 = (497779.69615306245 - 539589.7019980092) / 497779.69615306245 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.ContainsFalse&lt;Int32&gt;.LinkedList(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 26.0521828968254 > 25.328219790750005.
IsChangePoint: Marked as a change because one of 9/28/2022 6:40:13 PM, 10/22/2022 11:54:51 PM, 11/15/2022 8:48:03 PM, 11/22/2022 11:24:47 AM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -12.090409934144736 (T) = (0 -26414211.26329158) / Math.Sqrt((540445388590.58405 / (31)) + (209580473397.19965 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.07834797543710655 = (24495071.966527894 - 26414211.26329158) / 24495071.966527894 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.ContainsFalse&lt;Int32&gt;.ImmutableSortedSet(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 657.9031333333334 > 612.4709324768519.
IsChangePoint: Marked as a change because one of 9/28/2022 6:40:13 PM, 10/22/2022 11:54:51 PM, 11/15/2022 8:48:03 PM, 11/22/2022 11:24:47 AM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -9.698083935368434 (T) = (0 -629998.0985552319) / Math.Sqrt((242340606.7200309 / (31)) + (189760455.96949148 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.06305219711897628 = (592631.3874921824 - 629998.0985552319) / 592631.3874921824 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline b8584841ffed04ec8559629d0392eee9879f5dcb
Compare 59d56cc05b791bfb477ffc44cbef362287bd1054
Diff Diff

Regressions in System.Collections.IndexerSet<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SortedDictionary - Duration of single invocation 561.40 μs 606.03 μs 1.08 0.11 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.IndexerSet&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.IndexerSet<Int32>.SortedDictionary(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 606.0319205806213 > 587.1457360126203.
IsChangePoint: Marked as a change because one of 9/28/2022 6:40:13 PM, 10/22/2022 11:54:51 PM, 11/15/2022 8:48:03 PM, 11/22/2022 11:24:47 AM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -11.726594713570508 (T) = (0 -610967.1372434747) / Math.Sqrt((280472626.8033656 / (31)) + (154793328.04659733 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.07966924168592691 = (565883.6184769294 - 610967.1372434747) / 565883.6184769294 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline b8584841ffed04ec8559629d0392eee9879f5dcb
Compare 59d56cc05b791bfb477ffc44cbef362287bd1054
Diff Diff

Regressions in System.Collections.TryAddDefaultSize<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Dictionary - Duration of single invocation 96.57 μs 109.67 μs 1.14 0.01 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.TryAddDefaultSize&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.TryAddDefaultSize<Int32>.Dictionary(Count: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 109.67496327214454 > 101.46454335448757.
IsChangePoint: Marked as a change because one of 11/15/2022 8:48:03 PM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -35.165020986273696 (T) = (0 -109682.93613935493) / Math.Sqrt((2277527.5547315413 / (31)) + (1601815.7002851355 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.13228777751945336 = (96868.42719404919 - 109682.93613935493) / 96868.42719404919 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline b8584841ffed04ec8559629d0392eee9879f5dcb
Compare 59d56cc05b791bfb477ffc44cbef362287bd1054
Diff Diff

Regressions in System.Security.Cryptography.Tests.Perf_CryptoConfig

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
CreateFromName - Duration of single invocation 13.03 μs 15.48 μs 1.19 0.00 False
CreateFromName - Duration of single invocation 9.09 μs 11.68 μs 1.29 0.01 False
CreateFromName - Duration of single invocation 10.33 μs 12.98 μs 1.26 0.01 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Security.Cryptography.Tests.Perf_CryptoConfig*'

Payloads

Baseline
Compare

Histogram

System.Security.Cryptography.Tests.Perf_CryptoConfig.CreateFromName(name: "RSA")

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.478891380182002 > 13.772103825355629.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -43.50787001186446 (T) = (0 -15424.189846720663) / Math.Sqrt((43552.8373467554 / (31)) + (19695.523568056047 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.1498449239779351 = (13414.147877750378 - 15424.189846720663) / 13414.147877750378 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Security.Cryptography.Tests.Perf_CryptoConfig.CreateFromName(name: "X509Chain")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 11.682878313550939 > 9.811988663805593.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -45.01349894992459 (T) = (0 -11355.600106245125) / Math.Sqrt((15940.943435507324 / (31)) + (34532.66772388815 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.20172155774236997 = (9449.44361951738 - 11355.600106245125) / 9449.44361951738 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Security.Cryptography.Tests.Perf_CryptoConfig.CreateFromName(name: "SHA512")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 12.978774458919219 > 10.86105047789.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -44.176798725767654 (T) = (0 -12885.463538710752) / Math.Sqrt((49251.3704550721 / (31)) + (31670.91914600724 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.21976752243816228 = (10563.86836153362 - 12885.463538710752) / 10563.86836153362 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline b8584841ffed04ec8559629d0392eee9879f5dcb
Compare 59d56cc05b791bfb477ffc44cbef362287bd1054
Diff Diff

Regressions in JetStream.Poker

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Play - Duration of single invocation 337.54 ms 376.40 ms 1.12 0.01 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'JetStream.Poker*'

Payloads

Baseline
Compare

Histogram

JetStream.Poker.Play

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 376.3968116923077 > 357.957784425.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -32.03966116022152 (T) = (0 -377503515.16226435) / Math.Sqrt((26684617734412.848 / (31)) + (8202567964826.0625 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.10082603329971902 = (342927496.02832335 - 377503515.16226435) / 342927496.02832335 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline b8584841ffed04ec8559629d0392eee9879f5dcb
Compare 59d56cc05b791bfb477ffc44cbef362287bd1054
Diff Diff

Regressions in System.Collections.Sort<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
LinqQuery - Duration of single invocation 669.78 μs 722.79 μs 1.08 0.04 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Sort&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.Sort<Int32>.LinqQuery(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 722.7930888 > 705.4381956930001.
IsChangePoint: Marked as a change because one of 9/28/2022 6:40:13 PM, 10/22/2022 11:54:51 PM, 11/15/2022 8:48:03 PM, 11/22/2022 11:24:47 AM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -10.144533983540374 (T) = (0 -722078.8864201437) / Math.Sqrt((378565008.8694884 / (31)) + (85711351.98266946 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.058321933344790806 = (682286.6121068073 - 722078.8864201437) / 682286.6121068073 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline b8584841ffed04ec8559629d0392eee9879f5dcb
Compare 59d56cc05b791bfb477ffc44cbef362287bd1054
Diff Diff

Regressions in System.Numerics.Tests.Perf_Quaternion

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
EqualsBenchmark - Duration of single invocation 104.23 ns 116.46 ns 1.12 0.20 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_Quaternion*'

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_Quaternion.EqualsBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 116.45827654340209 > 108.8952436533352.
IsChangePoint: Marked as a change because one of 10/17/2022 12:47:34 PM, 11/15/2022 5:29:43 PM, 12/2/2022 12:08:23 PM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -6.847929955152428 (T) = (0 -118.59514711714644) / Math.Sqrt((72.59028543572707 / (31)) + (6.2880531422068335 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.10210911896478188 = (107.6074456479805 - 118.59514711714644) / 107.6074456479805 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline b8584841ffed04ec8559629d0392eee9879f5dcb
Compare 59d56cc05b791bfb477ffc44cbef362287bd1054
Diff Diff

Regressions in System.Collections.Tests.Perf_PriorityQueue<Guid, Guid>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
HeapSort - Duration of single invocation 7.79 μs 9.20 μs 1.18 0.02 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Tests.Perf_PriorityQueue&lt;Guid, Guid&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.Tests.Perf_PriorityQueue<Guid, Guid>.HeapSort(Size: 10)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.197941889345666 > 8.256499619721797.
IsChangePoint: Marked as a change because one of 11/15/2022 8:48:03 PM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -37.82077091355514 (T) = (0 -9050.15023026235) / Math.Sqrt((13843.57222675587 / (31)) + (11743.100111061505 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.1416504772990194 = (7927.251299954521 - 9050.15023026235) / 7927.251299954521 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline b8584841ffed04ec8559629d0392eee9879f5dcb
Compare 59d56cc05b791bfb477ffc44cbef362287bd1054
Diff Diff

Regressions in System.Formats.Cbor.Tests.Perf_CborReader

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SkipValue - Duration of single invocation 14.51 μs 17.23 μs 1.19 0.02 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Formats.Cbor.Tests.Perf_CborReader*'

Payloads

Baseline
Compare

Histogram

System.Formats.Cbor.Tests.Perf_CborReader.SkipValue(encoding: (Map, Strict))

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 17.225423444141178 > 15.13123997712594.
IsChangePoint: Marked as a change because one of 10/11/2022 6:23:40 AM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -36.811537120751375 (T) = (0 -16979.00703288266) / Math.Sqrt((83596.79743722717 / (31)) + (45358.32120438548 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.1674471470436891 = (14543.7051055188 - 16979.00703288266) / 14543.7051055188 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline b8584841ffed04ec8559629d0392eee9879f5dcb
Compare 59d56cc05b791bfb477ffc44cbef362287bd1054
Diff Diff

Regressions in System.Collections.AddGivenSize<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SortedList - Duration of single invocation 1.50 ms 2.28 ms 1.52 0.05 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.AddGivenSize&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.AddGivenSize<String>.SortedList(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.2799567602040813 > 1.5619375355539773.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -111.66001354217632 (T) = (0 -2267099.257631197) / Math.Sqrt((597933166.3845254 / (31)) + (746125055.9099427 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.5091571472866012 = (1502228.7517952274 - 2267099.257631197) / 1502228.7517952274 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline b8584841ffed04ec8559629d0392eee9879f5dcb
Compare 59d56cc05b791bfb477ffc44cbef362287bd1054
Diff Diff

Regressions in System.Collections.CopyTo<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ImmutableArray - Duration of single invocation 521.93 ns 2.19 μs 4.19 0.04 False
Array - Duration of single invocation 508.19 ns 2.16 μs 4.26 0.05 False
List - Duration of single invocation 377.93 ns 1.15 μs 3.03 0.04 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CopyTo&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.CopyTo<Int32>.ImmutableArray(Size: 2048)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.18554753265687 > 557.7358721604704.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -351.4981495908505 (T) = (0 -2169.019568342673) / Math.Sqrt((95.54197231722057 / (31)) + (502.2916293231674 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -3.0757420316893413 = (532.1778344846922 - 2169.019568342673) / 532.1778344846922 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CopyTo&lt;Int32&gt;.Array(Size: 2048)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.163671002964198 > 537.9080869327487.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -384.3875308542764 (T) = (0 -2153.317000020301) / Math.Sqrt((73.03021153419019 / (31)) + (426.27392478396 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -3.173091427232549 = (516.0004369825913 - 2153.317000020301) / 516.0004369825913 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CopyTo&lt;Int32&gt;.List(Size: 2048)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.1453248469238102 > 393.49806382024036.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -258.13681239910073 (T) = (0 -1135.708832959692) / Math.Sqrt((51.607785747521056 / (31)) + (188.55542065190485 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -2.015848199149501 = (376.5802381167504 - 1135.708832959692) / 376.5802381167504 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline b8584841ffed04ec8559629d0392eee9879f5dcb
Compare 59d56cc05b791bfb477ffc44cbef362287bd1054
Diff Diff

Regressions in System.Collections.CtorFromCollectionNonGeneric<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ArrayList - Duration of single invocation 1.50 μs 4.02 μs 2.69 0.02 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CtorFromCollectionNonGeneric&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.CtorFromCollectionNonGeneric<String>.ArrayList(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 4.017586423394768 > 1.5767228127570918.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -443.194572660888 (T) = (0 -4002.0163773099816) / Math.Sqrt((387.7725377140786 / (31)) + (513.2212164738577 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -1.6434750799313242 = (1513.9224907745117 - 4002.0163773099816) / 1513.9224907745117 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline b8584841ffed04ec8559629d0392eee9879f5dcb
Compare 59d56cc05b791bfb477ffc44cbef362287bd1054
Diff Diff

Regressions in System.Collections.Tests.Perf_Dictionary

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ContainsValue - Duration of single invocation 453.56 ms 485.48 ms 1.07 0.05 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Tests.Perf_Dictionary*'

Payloads

Baseline
Compare

Histogram

System.Collections.Tests.Perf_Dictionary.ContainsValue(Items: 3000)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 485.48278714285715 > 476.95150116384616.
IsChangePoint: Marked as a change because one of 9/28/2022 6:40:13 PM, 10/22/2022 11:54:51 PM, 11/15/2022 8:48:03 PM, 11/22/2022 11:24:47 AM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -9.594573551794767 (T) = (0 -481971922.03144073) / Math.Sqrt((126516615196715.11 / (31)) + (87753607978780.27 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.05697156850480324 = (455993270.2004846 - 481971922.03144073) / 455993270.2004846 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

Run Information

Architecture x64
OS ubuntu 18.04
Baseline b8584841ffed04ec8559629d0392eee9879f5dcb
Compare 59d56cc05b791bfb477ffc44cbef362287bd1054
Diff Diff

Regressions in System.Collections.CreateAddAndClear<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Dictionary - Duration of single invocation 162.50 μs 175.32 μs 1.08 0.02 False
Queue - Duration of single invocation 31.08 μs 42.35 μs 1.36 0.11 False
IDictionary - Duration of single invocation 165.01 μs 179.69 μs 1.09 0.01 False
ImmutableArray - Duration of single invocation 314.40 μs 1.16 ms 3.70 0.00 False
ConcurrentBag - Duration of single invocation 87.10 μs 91.65 μs 1.05 0.09 False
ICollection - Duration of single invocation 23.41 μs 34.65 μs 1.48 0.03 False
SortedList - Duration of single invocation 1.49 ms 2.40 ms 1.61 0.04 False
List - Duration of single invocation 16.57 μs 31.33 μs 1.89 0.09 False
Array - Duration of single invocation 8.40 μs 9.10 μs 1.08 0.13 False
HashSet - Duration of single invocation 150.87 μs 164.97 μs 1.09 0.02 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CreateAddAndClear&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.CreateAddAndClear<String>.Dictionary(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 175.32218794642858 > 171.31239210425727.
IsChangePoint: Marked as a change because one of 11/15/2022 8:48:03 PM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -20.84343670402827 (T) = (0 -177172.46605604174) / Math.Sqrt((5712323.722329607 / (31)) + (5940913.778011317 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.08085234089069998 = (163919.2138956177 - 177172.46605604174) / 163919.2138956177 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndClear&lt;String&gt;.Queue(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 42.351101702997276 > 31.889366362171693.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -35.11390435649486 (T) = (0 -42892.434711685455) / Math.Sqrt((1755403.4082171167 / (31)) + (1515601.2754134657 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.3791199562694824 = (31101.308132549566 - 42892.434711685455) / 31101.308132549566 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndClear&lt;String&gt;.IDictionary(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 179.68695945881223 > 172.7714455804893.
IsChangePoint: Marked as a change because one of 11/15/2022 8:48:03 PM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -24.811641740140097 (T) = (0 -179250.79485754133) / Math.Sqrt((4673539.888789085 / (31)) + (3836472.9230737286 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.08097145619296071 = (165823.80027761238 - 179250.79485754133) / 165823.80027761238 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndClear&lt;String&gt;.ImmutableArray(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.1641715360863094 > 334.39639942723215.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -303.6313468708617 (T) = (0 -1159547.6622787241) / Math.Sqrt((35513650.679409616 / (31)) + (175937378.61335593 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -2.6338365408783524 = (319097.3642414978 - 1159547.6622787241) / 319097.3642414978 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndClear&lt;String&gt;.ConcurrentBag(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 91.65329422757473 > 85.4399047420536.
IsChangePoint: Marked as a change because one of 11/15/2022 8:48:03 PM, 12/2/2022 5:21:30 AM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -13.00636962546323 (T) = (0 -90598.44004474692) / Math.Sqrt((9135057.91557162 / (31)) + (2002341.5365895615 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.09551528365523969 = (82699.38484332309 - 90598.44004474692) / 82699.38484332309 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndClear&lt;String&gt;.ICollection(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 34.65400104166667 > 24.206115226976177.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -75.5992480918137 (T) = (0 -34597.84864031683) / Math.Sqrt((401193.39443277713 / (31)) + (272985.5681978999 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.49647086879876984 = (23119.62722541256 - 34597.84864031683) / 23119.62722541256 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndClear&lt;String&gt;.SortedList(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.40324301577381 > 1.5874990202196242.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -87.24413655571172 (T) = (0 -2314087.295627713) / Math.Sqrt((867494670.2945462 / (31)) + (1515247723.424618 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.5284725640483479 = (1513986.8062129738 - 2314087.295627713) / 1513986.8062129738 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndClear&lt;String&gt;.List(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 31.334673161080072 > 17.399197281020435.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -47.49592318820753 (T) = (0 -28746.62974887967) / Math.Sqrt((64177.52566729354 / (31)) + (1666965.4099665831 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.7163311649951896 = (16748.882928406325 - 28746.62974887967) / 16748.882928406325 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndClear&lt;String&gt;.Array(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.09880795987112 > 8.812676019034198.
IsChangePoint: Marked as a change because one of 10/24/2022 10:11:47 PM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -6.26869106344746 (T) = (0 -9082.54627555117) / Math.Sqrt((105424.33343130047 / (31)) + (134917.46330150854 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.0675189606320061 = (8508.08895251285 - 9082.54627555117) / 8508.08895251285 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndClear&lt;String&gt;.HashSet(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 164.9731662371134 > 155.3386611162285.
IsChangePoint: Marked as a change because one of 9/28/2022 6:40:13 PM, 11/15/2022 5:29:43 PM, 12/2/2022 5:21:30 AM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -18.590937242445094 (T) = (0 -161932.76765867678) / Math.Sqrt((8620785.929770675 / (31)) + (4794421.561015721 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.08400786064691053 = (149383.38875332425 - 161932.76765867678) / 149383.38875332425 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline b8584841ffed04ec8559629d0392eee9879f5dcb
Compare 59d56cc05b791bfb477ffc44cbef362287bd1054
Diff Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Ctor - Duration of single invocation 87.12 μs 92.62 μs 1.06 0.00 False
Ctor - Duration of single invocation 208.07 μs 219.87 μs 1.06 0.00 False
Ctor - Duration of single invocation 86.29 μs 92.55 μs 1.07 0.00 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas*'

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas.Ctor(Pattern: "[\w]+://[^/\\s?#]+[^\\s?#]+(?:\?[^\\s#])?(?:#[^\\s])?", Options: None)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 92.6201729976754 > 91.32874277376763.
IsChangePoint: Marked as a change because one of 11/15/2022 8:48:03 PM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -24.217869019436463 (T) = (0 -92917.5592963161) / Math.Sqrt((1099234.5080575952 / (31)) + (422196.8241797617 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.06260420837527596 = (87443.24421450132 - 92917.5592963161) / 87443.24421450132 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas.Ctor(Pattern: "(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9])\\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9])", Options: Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 219.87356073943656 > 218.27748862154604.
IsChangePoint: Marked as a change because one of 11/15/2022 8:48:03 PM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -26.404890067628475 (T) = (0 -220189.65623264608) / Math.Sqrt((3940380.0957020177 / (31)) + (1222347.0097370767 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.052397699919852624 = (209226.66046249916 - 220189.65623264608) / 209226.66046249916 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas.Ctor(Pattern: "[\\w]+://[^/\\s?#]+[^\\s?#]+(?:\\?[^\\s#]*)?(?:#[^\\s]*)?", Options: Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 92.54848833636778 > 91.45390108692344.
IsChangePoint: Marked as a change because one of 11/15/2022 8:48:03 PM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -24.076681046474075 (T) = (0 -93076.23471318344) / Math.Sqrt((1151360.2697199318 / (31)) + (372991.3048195606 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.06201283811834843 = (87641.34610471745 - 93076.23471318344) / 87641.34610471745 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline b8584841ffed04ec8559629d0392eee9879f5dcb
Compare 59d56cc05b791bfb477ffc44cbef362287bd1054
Diff Diff

Regressions in System.Collections.Tests.Perf_PriorityQueue<String, String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
HeapSort - Duration of single invocation 17.07 μs 18.46 μs 1.08 0.04 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Tests.Perf_PriorityQueue&lt;String, String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.Tests.Perf_PriorityQueue<String, String>.HeapSort(Size: 10)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 18.461768769817677 > 18.127465237903436.
IsChangePoint: Marked as a change because one of 10/11/2022 5:13:36 PM, 11/16/2022 9:45:14 AM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -15.160411786991228 (T) = (0 -18536.069062619154) / Math.Sqrt((122134.6415019355 / (31)) + (69936.67656815276 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.07076989945557553 = (17310.973227808954 - 18536.069062619154) / 17310.973227808954 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline b8584841ffed04ec8559629d0392eee9879f5dcb
Compare 59d56cc05b791bfb477ffc44cbef362287bd1054
Diff Diff

Regressions in System.Collections.Tests.DictionarySequentialKeys

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
TryGetValue_3k_Int_Int - Duration of single invocation 184.40 ns 194.51 ns 1.05 0.07 False
ContainsKey_3k_Int_32ByteValue - Duration of single invocation 179.02 ns 193.14 ns 1.08 0.07 False
ContainsKey_17_Int_32ByteValue - Duration of single invocation 179.51 ns 193.10 ns 1.08 0.11 False
ContainsKey_3k_Int_32ByteRefsValue - Duration of single invocation 179.02 ns 190.59 ns 1.06 0.10 False
ContainsKey_17_Int_32ByteRefsValue - Duration of single invocation 184.67 ns 197.01 ns 1.07 0.07 False
ContainsValue_3k_Int_Int - Duration of single invocation 181.52 ns 192.13 ns 1.06 0.07 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Tests.DictionarySequentialKeys*'

Payloads

Baseline
Compare

Histogram

System.Collections.Tests.DictionarySequentialKeys.TryGetValue_3k_Int_Int

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 194.5139038249559 > 192.18292218750003.
IsChangePoint: Marked as a change because one of 10/22/2022 11:54:51 PM, 11/15/2022 8:48:03 PM, 12/2/2022 5:21:30 AM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -7.364873512619879 (T) = (0 -198.52145192137598) / Math.Sqrt((46.417156232429186 / (31)) + (28.637597506788346 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.06307700953472226 = (186.7423057227651 - 198.52145192137598) / 186.7423057227651 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Tests.DictionarySequentialKeys.ContainsKey_3k_Int_32ByteValue

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 193.1362388117284 > 187.93731159662357.
IsChangePoint: Marked as a change because one of 10/22/2022 11:54:51 PM, 11/15/2022 8:48:03 PM, 12/2/2022 5:21:30 AM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -8.209524560651934 (T) = (0 -196.00738832303549) / Math.Sqrt((42.201985714953594 / (31)) + (34.99876517101591 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.07328332352041582 = (182.6240881858881 - 196.00738832303549) / 182.6240881858881 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Tests.DictionarySequentialKeys.ContainsKey_17_Int_32ByteValue

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 193.09595991735185 > 188.24833667118068.
IsChangePoint: Marked as a change because one of 10/22/2022 11:54:51 PM, 11/15/2022 8:48:03 PM, 12/2/2022 5:21:30 AM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -6.850515251949422 (T) = (0 -198.3852607475725) / Math.Sqrt((41.923941105934745 / (31)) + (70.77837297456777 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.0739249648407187 = (184.72916380801 - 198.3852607475725) / 184.72916380801 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Tests.DictionarySequentialKeys.ContainsKey_3k_Int_32ByteRefsValue

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 190.587727148623 > 188.0535298221983.
IsChangePoint: Marked as a change because one of 10/22/2022 11:54:51 PM, 11/15/2022 8:48:03 PM, 12/2/2022 5:21:30 AM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -9.69803357599574 (T) = (0 -194.4751344715037) / Math.Sqrt((26.608866880292457 / (31)) + (19.443706481273445 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.06684050410264424 = (182.29073017346988 - 194.4751344715037) / 182.29073017346988 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Tests.DictionarySequentialKeys.ContainsKey_17_Int_32ByteRefsValue

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 197.00776183819926 > 188.85408822260072.
IsChangePoint: Marked as a change because one of 10/22/2022 11:54:51 PM, 11/15/2022 8:48:03 PM, 12/2/2022 5:21:30 AM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -8.813174957717234 (T) = (0 -198.15554313780376) / Math.Sqrt((43.55913785364213 / (31)) + (27.812588265249538 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.07458230195133426 = (184.40238851688983 - 198.15554313780376) / 184.40238851688983 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Tests.DictionarySequentialKeys.ContainsValue_3k_Int_Int

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 192.1298373100665 > 189.30445561961204.
IsChangePoint: Marked as a change because one of 10/22/2022 11:54:51 PM, 11/15/2022 8:48:03 PM, 12/2/2022 5:21:30 AM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -8.066683217505567 (T) = (0 -195.32835275543928) / Math.Sqrt((37.62825928922754 / (31)) + (35.70970265531359 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.07040203356270555 = (182.48129827006414 - 195.32835275543928) / 182.48129827006414 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline b8584841ffed04ec8559629d0392eee9879f5dcb
Compare 59d56cc05b791bfb477ffc44cbef362287bd1054
Diff Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Common

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
CtorInvoke - Duration of single invocation 115.63 μs 122.92 μs 1.06 0.00 False
CtorInvoke - Duration of single invocation 179.76 μs 200.54 μs 1.12 0.00 False
Ctor - Duration of single invocation 113.71 μs 120.34 μs 1.06 0.00 False
CtorInvoke - Duration of single invocation 116.10 μs 123.35 μs 1.06 0.00 False
Ctor - Duration of single invocation 176.78 μs 197.17 μs 1.12 0.00 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Common*'

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Common.CtorInvoke(Options: None)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 122.92246574519231 > 121.21140022288603.
IsChangePoint: Marked as a change because one of 9/28/2022 6:40:13 PM, 11/15/2022 8:48:03 PM, 12/2/2022 5:21:30 AM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -21.90986872908947 (T) = (0 -123244.47457870106) / Math.Sqrt((2225455.867500732 / (31)) + (488427.42537746194 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.05629724005593348 = (116675.94111310465 - 123244.47457870106) / 116675.94111310465 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.CtorInvoke(Options: IgnoreCase, Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 200.5359862713675 > 188.9545887661638.
IsChangePoint: Marked as a change because one of 9/28/2022 6:40:13 PM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -33.410741696297634 (T) = (0 -199962.3601925013) / Math.Sqrt((8385618.782251457 / (31)) + (1006904.629966811 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.10216869347344842 = (181426.27474050864 - 199962.3601925013) / 181426.27474050864 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Ctor(Options: Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 120.3419048187023 > 119.12599654072606.
IsChangePoint: Marked as a change because one of 9/28/2022 6:40:13 PM, 11/15/2022 8:48:03 PM, 12/2/2022 5:21:30 AM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -18.56692792397002 (T) = (0 -120519.40888710279) / Math.Sqrt((2815749.169661377 / (31)) + (627753.3765048056 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.05489042325073609 = (114248.2728354968 - 120519.40888710279) / 114248.2728354968 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.CtorInvoke(Options: Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 123.34678213582677 > 121.53638394756776.
IsChangePoint: Marked as a change because one of 9/28/2022 6:40:13 PM, 11/15/2022 8:48:03 PM, 12/2/2022 5:21:30 AM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -20.665590019798373 (T) = (0 -123221.34618794319) / Math.Sqrt((2226282.6996742524 / (31)) + (682033.2317120985 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.05513481025950727 = (116782.56180140362 - 123221.34618794319) / 116782.56180140362 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Ctor(Options: IgnoreCase, Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 197.17460567708335 > 186.44349306107955.
IsChangePoint: Marked as a change because one of 9/28/2022 6:40:13 PM, 11/15/2022 8:48:03 PM, 12/2/2022 5:21:30 AM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -29.99139458258897 (T) = (0 -197376.09220489746) / Math.Sqrt((10039513.232697079 / (31)) + (1088399.5136831054 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.10095340336438927 = (179277.42591261212 - 197376.09220489746) / 179277.42591261212 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline b8584841ffed04ec8559629d0392eee9879f5dcb
Compare 59d56cc05b791bfb477ffc44cbef362287bd1054
Diff Diff

Regressions in System.Collections.CtorFromCollectionNonGeneric<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ArrayList - Duration of single invocation 138.80 μs 538.13 μs 3.88 0.02 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CtorFromCollectionNonGeneric&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.CtorFromCollectionNonGeneric<Int32>.ArrayList(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 538.1325524553572 > 144.76865245476978.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -293.0434162248978 (T) = (0 -530544.9746768988) / Math.Sqrt((5746883.226050559 / (31)) + (43175689.44808106 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -2.814557590412708 = (139084.27441503055 - 530544.9746768988) / 139084.27441503055 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler performanceautofiler bot added refs/heads/main untriaged New issue has not been triaged by the area owner labels Dec 13, 2022
@kotlarmilos
Copy link
Member

It looks to me that all regressions are related to a94fa06.

Other relevant change was 0116323.

@dotnet-issue-labeler
Copy link

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

@kotlarmilos kotlarmilos transferred this issue from dotnet/perf-autofiling-issues Dec 13, 2022
@kotlarmilos kotlarmilos added area-Codegen-Interpreter-mono and removed untriaged New issue has not been triaged by the area owner labels Dec 13, 2022
@kotlarmilos kotlarmilos added this to the 8.0.0 milestone Dec 13, 2022
@ghost
Copy link

ghost commented Dec 13, 2022

Tagging subscribers to this area: @BrzVlad
See info in area-owners.md if you want to be subscribed.

Issue Details

Run Information

Architecture x64
OS ubuntu 18.04
Baseline b8584841ffed04ec8559629d0392eee9879f5dcb
Compare 59d56cc05b791bfb477ffc44cbef362287bd1054
Diff Diff

Regressions in System.Collections.TryGetValueFalse<Int32, Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SortedDictionary - Duration of single invocation 607.25 μs 692.09 μs 1.14 0.14 False
ImmutableSortedDictionary - Duration of single invocation 586.01 μs 648.61 μs 1.11 0.05 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.TryGetValueFalse&lt;Int32, Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.TryGetValueFalse<Int32, Int32>.SortedDictionary(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 692.0862336956523 > 653.6626083312499.
IsChangePoint: Marked as a change because one of 9/28/2022 6:40:13 PM, 10/22/2022 11:54:51 PM, 11/15/2022 8:48:03 PM, 11/22/2022 11:24:47 AM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -11.19507384735334 (T) = (0 -689448.823493863) / Math.Sqrt((484235957.5897148 / (31)) + (249520970.04535538 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.08809694114354276 = (633628.1239512374 - 689448.823493863) / 633628.1239512374 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.TryGetValueFalse&lt;Int32, Int32&gt;.ImmutableSortedDictionary(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 648.6063610491071 > 624.197248987611.
IsChangePoint: Marked as a change because one of 9/28/2022 3:03:11 PM, 10/22/2022 11:54:51 PM, 11/15/2022 8:48:03 PM, 11/22/2022 2:57:03 PM, 12/7/2022 10:10:38 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -9.201833885530117 (T) = (0 -641639.1238436721) / Math.Sqrt((252093372.5097592 / (32)) + (245576912.4583364 / (26))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (26) - 2, .025) and -0.06347841706644324 = (603340.0523666521 - 641639.1238436721) / 603340.0523666521 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline b8584841ffed04ec8559629d0392eee9879f5dcb
Compare 59d56cc05b791bfb477ffc44cbef362287bd1054
Diff Diff

Regressions in Benchstone.MDBenchI.MDGeneralArray

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Test - Duration of single invocation 938.11 ms 6.29 secs 6.71 0.15 False
Test2 - Duration of single invocation 934.69 ms 6.31 secs 6.75 0.12 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Benchstone.MDBenchI.MDGeneralArray*'

Payloads

Baseline
Compare

Histogram

Benchstone.MDBenchI.MDGeneralArray.Test

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 6.294134878285714 > 974.7867649853847.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -332.67252279848014 (T) = (0 -6324614195.630578) / Math.Sqrt((906468370819308.8 / (31)) + (6261031663079951 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -5.666072090617283 = (948776747.3341132 - 6324614195.630578) / 948776747.3341132 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### Benchstone.MDBenchI.MDGeneralArray.Test2

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 6.309850924928572 > 979.834198875.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -341.4035925796587 (T) = (0 -6285254191.324551) / Math.Sqrt((714591220814518.2 / (31)) + (5974629693366718 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -5.625040301109063 = (948711842.5939794 - 6285254191.324551) / 948711842.5939794 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline b8584841ffed04ec8559629d0392eee9879f5dcb
Compare 59d56cc05b791bfb477ffc44cbef362287bd1054
Diff Diff

Regressions in System.Collections.CreateAddAndClear<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
List - Duration of single invocation 13.03 μs 23.47 μs 1.80 0.07 False
HashSet - Duration of single invocation 74.27 μs 86.80 μs 1.17 0.05 False
Array - Duration of single invocation 5.16 μs 5.72 μs 1.11 0.04 False
SortedSet - Duration of single invocation 586.86 μs 640.27 μs 1.09 0.05 False
ImmutableArray - Duration of single invocation 261.26 μs 1.09 ms 4.17 0.02 False
ConcurrentBag - Duration of single invocation 72.90 μs 83.50 μs 1.15 0.06 False
IDictionary - Duration of single invocation 101.90 μs 115.97 μs 1.14 0.01 False
Queue - Duration of single invocation 25.43 μs 37.61 μs 1.48 0.11 False
SortedList - Duration of single invocation 452.70 μs 1.24 ms 2.73 0.00 False
Dictionary - Duration of single invocation 99.86 μs 111.87 μs 1.12 0.02 False
ICollection - Duration of single invocation 17.96 μs 29.35 μs 1.63 0.06 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CreateAddAndClear&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.CreateAddAndClear<Int32>.List(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 23.469186698893516 > 12.916729737048087.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -161.81748973999427 (T) = (0 -23729.039701293033) / Math.Sqrt((97733.2729470524 / (31)) + (45991.36346664534 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.9055328450654487 = (12452.70568950913 - 23729.039701293033) / 12452.70568950913 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndClear&lt;Int32&gt;.HashSet(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 86.8032005704365 > 78.426903206159.
IsChangePoint: Marked as a change because one of 9/23/2022 1:16:35 AM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -36.40434044961095 (T) = (0 -86958.13508971513) / Math.Sqrt((2217138.4295794037 / (31)) + (824857.8523305526 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.15440119547874265 = (75327.48184105323 - 86958.13508971513) / 75327.48184105323 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndClear&lt;Int32&gt;.Array(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 5.716856630642543 > 5.399100894675782.
IsChangePoint: Marked as a change because one of 11/15/2022 8:48:03 PM, 12/2/2022 5:21:30 AM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -18.48518076366622 (T) = (0 -5567.102932469006) / Math.Sqrt((6380.931601219815 / (31)) + (6821.039358783926 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.07653819458003623 = (5171.300898098432 - 5567.102932469006) / 5171.300898098432 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndClear&lt;Int32&gt;.SortedSet(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 640.2655003720239 > 616.7127253262997.
IsChangePoint: Marked as a change because one of 9/28/2022 6:40:13 PM, 10/22/2022 11:54:51 PM, 11/15/2022 8:48:03 PM, 11/22/2022 11:24:47 AM, 12/8/2022 3:49:24 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -12.01220410996348 (T) = (0 -643854.4397887815) / Math.Sqrt((398689911.64089644 / (36)) + (46558078.043755606 / (22))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (36) + (22) - 2, .025) and -0.07268521834314194 = (600226.8221643551 - 643854.4397887815) / 600226.8221643551 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndClear&lt;Int32&gt;.ImmutableArray(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.088982142857143 > 276.2522915484152.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -376.8357690560496 (T) = (0 -1090594.941926711) / Math.Sqrt((18660112.45729481 / (31)) + (113333022.77851015 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -3.1148981078260696 = (265035.71008296 - 1090594.941926711) / 265035.71008296 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndClear&lt;Int32&gt;.ConcurrentBag(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 83.5045912403475 > 76.60009291431061.
IsChangePoint: Marked as a change because one of 11/15/2022 8:48:03 PM, 12/2/2022 5:21:30 AM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -14.055342969139994 (T) = (0 -82297.57627283916) / Math.Sqrt((6139535.682714304 / (31)) + (2690300.2627694877 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.10275834182771805 = (74628.84038259795 - 82297.57627283916) / 74628.84038259795 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndClear&lt;Int32&gt;.IDictionary(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 115.9718737132353 > 107.65308224498571.
IsChangePoint: Marked as a change because one of 11/15/2022 8:48:03 PM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -25.98727133750973 (T) = (0 -115309.30339318319) / Math.Sqrt((3628105.640189915 / (31)) + (2778664.8968289616 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.11818768059609838 = (103121.60059902696 - 115309.30339318319) / 103121.60059902696 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndClear&lt;Int32&gt;.Queue(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 37.612352560240964 > 26.790697540098424.
IsChangePoint: Marked as a change because one of 11/15/2022 8:48:03 PM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -45.925900069924374 (T) = (0 -37014.780575339915) / Math.Sqrt((1434411.4025058036 / (31)) + (250737.15594331396 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.41333275150141474 = (26189.714018880757 - 37014.780575339915) / 26189.714018880757 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndClear&lt;Int32&gt;.SortedList(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.2351747310897436 > 481.12689615472993.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -301.61076696726565 (T) = (0 -1229816.142481173) / Math.Sqrt((55609320.484936915 / (31)) + (126180305.55516766 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -1.6573410010932557 = (462799.5210156373 - 1229816.142481173) / 462799.5210156373 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndClear&lt;Int32&gt;.Dictionary(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 111.87108105654761 > 104.90005048467356.
IsChangePoint: Marked as a change because one of 11/15/2022 10:02:59 PM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -39.77603222684615 (T) = (0 -112061.74309436271) / Math.Sqrt((1971209.5081188811 / (31)) + (634294.283447297 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.11699701161321148 = (100324.12077138746 - 112061.74309436271) / 100324.12077138746 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndClear&lt;Int32&gt;.ICollection(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 29.352911646110055 > 18.873768375699232.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -47.78328987979911 (T) = (0 -29802.996069191984) / Math.Sqrt((344930.27369587903 / (31)) + (1243141.1594205475 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.6216680859074272 = (18377.987658624545 - 29802.996069191984) / 18377.987658624545 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline b8584841ffed04ec8559629d0392eee9879f5dcb
Compare 59d56cc05b791bfb477ffc44cbef362287bd1054
Diff Diff

Regressions in System.Tests.Perf_Array

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ArrayRetrieve1D - Duration of single invocation 56.81 ms 92.47 ms 1.63 0.01 False
ArrayRetrieve3D - Duration of single invocation 79.96 ms 195.12 ms 2.44 0.01 False
ArrayAssign3D - Duration of single invocation 69.93 ms 171.79 ms 2.46 0.01 False
Clear - Duration of single invocation 802.50 ns 1.28 μs 1.60 0.04 False
ArrayCopy3D - Duration of single invocation 895.19 ns 2.51 μs 2.81 0.01 False
ArrayAssign1D - Duration of single invocation 54.61 ms 89.41 ms 1.64 0.01 False
ArrayAssign2D - Duration of single invocation 65.28 ms 132.76 ms 2.03 0.01 False
ArrayRetrieve2D - Duration of single invocation 67.16 ms 137.49 ms 2.05 0.01 False
ArrayCopy2D - Duration of single invocation 890.28 ns 2.52 μs 2.84 0.00 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Array*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Array.ArrayRetrieve1D

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 92.47110893333334 > 59.529360575000005.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -21.223216619456945 (T) = (0 -100087510.99728836) / Math.Sqrt((944226172326.4021 / (31)) + (107004198809894.9 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.7353657290510317 = (57675168.59516424 - 100087510.99728836) / 57675168.59516424 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Array.ArrayRetrieve3D

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 195.11992573333333 > 83.9455244775.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -54.40797583460208 (T) = (0 -202777833.6621286) / Math.Sqrt((1706486419546.1235 / (31)) + (135365602452713.36 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -1.5256843560719324 = (80286292.77234729 - 202777833.6621286) / 80286292.77234729 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Array.ArrayAssign3D

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 171.7920578 > 73.9266687490625.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -55.45285415461456 (T) = (0 -182884483.0767128) / Math.Sqrt((1110423131065.4656 / (31)) + (108905512668653.58 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -1.5750620889404228 = (71021387.74135944 - 182884483.0767128) / 71021387.74135944 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Array.Clear

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.2804935073141204 > 850.2766919782395.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -127.1820164510616 (T) = (0 -1281.975518770727) / Math.Sqrt((237.5497501364831 / (31)) + (146.7495100171736 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.5601731699878728 = (821.6879660741068 - 1281.975518770727) / 821.6879660741068 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Array.ArrayCopy3D

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.514441045422554 > 940.3953264838125.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -326.36903495578684 (T) = (0 -2511.4497234478554) / Math.Sqrt((92.51337904364681 / (31)) + (576.4641480733283 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -1.785979880695033 = (901.4601077525767 - 2511.4497234478554) / 901.4601077525767 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Array.ArrayAssign1D

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 89.40596875 > 57.93329311.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -21.44143410887711 (T) = (0 -97473520.43416937) / Math.Sqrt((760556966515.2164 / (31)) + (103192246062875.11 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.7587614009574852 = (55421685.04556896 - 97473520.43416937) / 55421685.04556896 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Array.ArrayAssign2D

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 132.7614871153846 > 67.79270777999999.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -40.95992139453779 (T) = (0 -142767105.53070477) / Math.Sqrt((921759610052.5035 / (31)) + (96164588354940.81 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -1.1915565594332844 = (65144157.43284443 - 142767105.53070477) / 65144157.43284443 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Array.ArrayRetrieve2D

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 137.49438343333333 > 70.714509425.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -38.38776010535931 (T) = (0 -144161378.61725006) / Math.Sqrt((573856769846.9851 / (31)) + (107603613011374.25 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -1.14050916337911 = (67349106.03684126 - 144161378.61725006) / 67349106.03684126 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Array.ArrayCopy2D

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.5241001749029435 > 937.8620140077794.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -324.6576433641876 (T) = (0 -2498.1992456941894) / Math.Sqrt((178.34028851076172 / (31)) + (501.1030098691688 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -1.7838377726305519 = (897.3939754160128 - 2498.1992456941894) / 897.3939754160128 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline b8584841ffed04ec8559629d0392eee9879f5dcb
Compare 59d56cc05b791bfb477ffc44cbef362287bd1054
Diff Diff

Regressions in System.Collections.CreateAddAndRemove<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SortedList - Duration of single invocation 2.67 ms 4.41 ms 1.65 0.02 False
Queue - Duration of single invocation 51.88 μs 64.29 μs 1.24 0.09 False
List - Duration of single invocation 167.39 μs 570.13 μs 3.41 0.05 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CreateAddAndRemove&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.CreateAddAndRemove<String>.SortedList(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 4.4123604302083335 > 2.8308888640625005.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -85.38138212497053 (T) = (0 -4277671.000971078) / Math.Sqrt((5664652178.633984 / (31)) + (4049769195.1352 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.5725245280355129 = (2720257.0927876006 - 4277671.000971078) / 2720257.0927876006 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndRemove&lt;String&gt;.Queue(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 64.28818438333332 > 54.06705439008348.
IsChangePoint: Marked as a change because one of 9/28/2022 9:30:44 PM, 11/15/2022 8:48:03 PM, 12/2/2022 5:21:30 AM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -24.59478322082287 (T) = (0 -63755.51149809263) / Math.Sqrt((4958665.781557593 / (31)) + (1047886.1980280007 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.20771189180473193 = (52790.33180903786 - 63755.51149809263) / 52790.33180903786 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndRemove&lt;String&gt;.List(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 570.1297382716049 > 175.73679156375115.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -260.4511859800968 (T) = (0 -560716.7040482993) / Math.Sqrt((16396587.887146804 / (31)) + (46053772.84641648 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -2.2718187228020907 = (171377.68059719505 - 560716.7040482993) / 171377.68059719505 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author: performanceautofiler[bot]
Assignees: vargaz
Labels:

area-Codegen-Interpreter-mono, refs/heads/main, ubuntu 18.04, Regression, x64
Milestone: -

This was referenced Dec 13, 2022
Closed
Closed
@vargaz
Copy link
Contributor

vargaz commented Dec 14, 2022

The Benchstone.MDBenchI.MDGeneralArray regression was definitely caused by a94fa06.

@vargaz vargaz mentioned this issue Dec 14, 2022
Merged
@jeffhandley jeffhandley added os-linux Linux OS (any supported distro) arch-x64 and removed refs/heads/main labels Dec 28, 2022
@ghost ghost locked as resolved and limited conversation to collaborators Jan 27, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@vargaz vargaz

Labels
arch-x64 area-Codegen-Interpreter-mono os-linux Linux OS (any supported distro)
Projects
None yet
Milestone
8.0.0
Development

No branches or pull requests
3 participants
@vargaz @jeffhandley @kotlarmilos