Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Linux/x64: 15 Regressions on 12/1/2022 11:48:13 PM #10359

Closed
performanceautofiler bot opened this issue Dec 6, 2022 · 1 comment
Closed

[Perf] Linux/x64: 15 Regressions on 12/1/2022 11:48:13 PM #10359

performanceautofiler bot opened this issue Dec 6, 2022 · 1 comment
Labels
arch-x64 branch-refs/heads/main kind-micro_mono mono-aot os-linux perf-regression runtime-mono

Comments

@performanceautofiler
Copy link

performanceautofiler bot commented Dec 6, 2022 •
edited
Loading

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 776a87d9f0c9105c2729d1ab781d74bffa2a182b
Compare 9d7ffb5b0f9c56376460a0a45b90cb3ca90302ea
Diff Diff

Regressions in System.Tests.Perf_Version

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Ctor4 - Duration of single invocation 10.34 ns 23.74 ns 2.30 0.18 False
Ctor3 - Duration of single invocation 10.26 ns 21.88 ns 2.13 0.16 False
TryParse2 - Duration of single invocation 88.38 ns 94.25 ns 1.07 0.08 False
Ctor2 - Duration of single invocation 10.61 ns 13.57 ns 1.28 0.16 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Version*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Version.Ctor4

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 23.737197908540285 > 10.84910095227404.
IsChangePoint: Marked as a change because one of 12/1/2022 7:29:59 PM, 12/6/2022 5:44:36 AM falls between 11/27/2022 4:37:47 PM and 12/6/2022 5:44:36 AM.
IsRegressionStdDev: Marked as regression because -75.36266270063606 (T) = (0 -23.39314609864366) / Math.Sqrt((0.02497006131500911 / (31)) + (0.5521112351335996 / (19))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (19) - 2, .025) and -1.2559402474991364 = (10.369576997695997 - 23.39314609864366) / 10.369576997695997 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Version.Ctor3

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 21.8806179799736 > 10.826767506030404.
IsChangePoint: Marked as a change because one of 12/1/2022 7:29:59 PM, 12/6/2022 5:44:36 AM falls between 11/27/2022 4:37:47 PM and 12/6/2022 5:44:36 AM.
IsRegressionStdDev: Marked as regression because -114.73973560892486 (T) = (0 -21.55701184195674) / Math.Sqrt((0.0505130687868358 / (31)) + (0.14691505696190077 / (19))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (19) - 2, .025) and -1.0618472039235212 = (10.45519367338936 - 21.55701184195674) / 10.45519367338936 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Version.TryParse2

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 94.25128474155282 > 93.23027667006441.
IsChangePoint: Marked as a change because one of 9/22/2022 3:11:11 PM, 12/1/2022 7:29:59 PM, 12/6/2022 5:44:36 AM falls between 11/27/2022 4:37:47 PM and 12/6/2022 5:44:36 AM.
IsRegressionStdDev: Marked as regression because -11.764135915181694 (T) = (0 -93.38216749324913) / Math.Sqrt((0.9105379569635996 / (31)) + (3.0791832456234696 / (18))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (18) - 2, .025) and -0.05977210046599515 = (88.11532918463112 - 93.38216749324913) / 88.11532918463112 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Version.Ctor2

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.566668404972763 > 11.132118159336619.
IsChangePoint: Marked as a change because one of 12/1/2022 7:29:59 PM, 12/6/2022 5:44:36 AM falls between 11/27/2022 4:37:47 PM and 12/6/2022 5:44:36 AM.
IsRegressionStdDev: Marked as regression because -30.376742145831525 (T) = (0 -13.931863532652988) / Math.Sqrt((0.1278935684198609 / (31)) + (0.1380452358161859 / (19))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (19) - 2, .025) and -0.3032886401119919 = (10.689775928266988 - 13.931863532652988) / 10.689775928266988 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline 776a87d9f0c9105c2729d1ab781d74bffa2a182b
Compare 9d7ffb5b0f9c56376460a0a45b90cb3ca90302ea
Diff Diff

Regressions in System.Tests.Perf_String

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Remove_IntInt - Duration of single invocation 31.97 ns 37.35 ns 1.17 0.05 False
Remove_IntInt - Duration of single invocation 32.08 ns 38.68 ns 1.21 0.07 False
Remove_IntInt - Duration of single invocation 36.61 ns 42.06 ns 1.15 0.07 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_String*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_String.Remove_IntInt(s: "dzsdzsDDZSDZSDZSddsz", i1: 0, i2: 8)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 37.35395499451654 > 33.59263740556034.
IsChangePoint: Marked as a change because one of 9/29/2022 8:01:59 PM, 12/1/2022 7:29:59 PM, 12/6/2022 5:44:36 AM falls between 11/27/2022 4:37:47 PM and 12/6/2022 5:44:36 AM.
IsRegressionStdDev: Marked as regression because -34.324193517562556 (T) = (0 -37.88581018702641) / Math.Sqrt((0.6098174741313332 / (31)) + (0.12544799630741948 / (19))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (19) - 2, .025) and -0.17213304248005845 = (32.322107486079986 - 37.88581018702641) / 32.322107486079986 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.Remove_IntInt(s: "dzsdzsDDZSDZSDZSddsz", i1: 7, i2: 4)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 38.68441360804141 > 33.69070961651581.
IsChangePoint: Marked as a change because one of 9/29/2022 8:01:59 PM, 12/1/2022 7:29:59 PM, 12/6/2022 5:44:36 AM falls between 11/27/2022 4:37:47 PM and 12/6/2022 5:44:36 AM.
IsRegressionStdDev: Marked as regression because -30.28457691832061 (T) = (0 -38.381093915971235) / Math.Sqrt((0.1544598251021003 / (31)) + (0.6959868893090417 / (19))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (19) - 2, .025) and -0.19183994076253288 = (32.20322847329249 - 38.381093915971235) / 32.20322847329249 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.Remove_IntInt(s: "dzsdzsDDZSDZSDZSddsz", i1: 10, i2: 1)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 42.05855311305282 > 38.869941714755605.
IsChangePoint: Marked as a change because one of 9/29/2022 8:01:59 PM, 12/1/2022 7:29:59 PM, 12/6/2022 5:44:36 AM falls between 11/27/2022 4:37:47 PM and 12/6/2022 5:44:36 AM.
IsRegressionStdDev: Marked as regression because -32.28836924431987 (T) = (0 -42.89083538967325) / Math.Sqrt((0.19167317554225158 / (31)) + (0.5165227631997725 / (19))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (19) - 2, .025) and -0.15944016332278765 = (36.992711436487006 - 42.89083538967325) / 36.992711436487006 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 776a87d9f0c9105c2729d1ab781d74bffa2a182b
Compare 9d7ffb5b0f9c56376460a0a45b90cb3ca90302ea
Diff Diff

Regressions in PerfLabTests.BlockCopyPerf

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
CallBlockCopy - Duration of single invocation 8.06 ns 21.34 ns 2.65 0.14 False
CallBlockCopy - Duration of single invocation 40.31 ns 45.45 ns 1.13 0.05 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'PerfLabTests.BlockCopyPerf*'

Payloads

Baseline
Compare

Histogram

PerfLabTests.BlockCopyPerf.CallBlockCopy(numElements: 10)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 21.34348301070222 > 8.461458956286068.
IsChangePoint: Marked as a change because one of 12/1/2022 7:29:59 PM, 12/6/2022 5:44:36 AM falls between 11/27/2022 4:37:47 PM and 12/6/2022 5:44:36 AM.
IsRegressionStdDev: Marked as regression because -320.34924337029776 (T) = (0 -21.31594712072687) / Math.Sqrt((0.025175877865653378 / (31)) + (0.016746372696761387 / (19))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (19) - 2, .025) and -1.6209741581532526 = (8.132833761224934 - 21.31594712072687) / 8.132833761224934 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### PerfLabTests.BlockCopyPerf.CallBlockCopy(numElements: 100)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 45.44766272339463 > 42.70463324224116.
IsChangePoint: Marked as a change because one of 12/1/2022 7:29:59 PM, 12/6/2022 5:44:36 AM falls between 11/27/2022 4:37:47 PM and 12/6/2022 5:44:36 AM.
IsRegressionStdDev: Marked as regression because -57.71045499588069 (T) = (0 -45.99684717872979) / Math.Sqrt((0.13759503379392182 / (31)) + (0.08347134934534471 / (19))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (19) - 2, .025) and -0.13367112415044705 = (40.57336047365501 - 45.99684717872979) / 40.57336047365501 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 776a87d9f0c9105c2729d1ab781d74bffa2a182b
Compare 9d7ffb5b0f9c56376460a0a45b90cb3ca90302ea
Diff Diff

Regressions in System.Numerics.Tests.Perf_Matrix4x4

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
CreatePerspectiveBenchmark - Duration of single invocation 30.21 ns 38.11 ns 1.26 0.01 False
CreatePerspectiveFieldOfViewBenchmark - Duration of single invocation 48.45 ns 59.82 ns 1.23 0.08 False
CreateScaleFromScalarWithCenterBenchmark - Duration of single invocation 33.14 ns 61.04 ns 1.84 0.00 False
CreatePerspectiveOffCenterBenchmark - Duration of single invocation 31.79 ns 39.47 ns 1.24 0.01 False
CreateScaleFromVectorWithCenterBenchmark - Duration of single invocation 37.75 ns 65.42 ns 1.73 0.01 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_Matrix4x4*'

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_Matrix4x4.CreatePerspectiveBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 38.11110776227614 > 31.744333212841294.
IsChangePoint: Marked as a change because one of 12/1/2022 7:29:59 PM, 12/6/2022 5:44:36 AM falls between 11/27/2022 4:37:47 PM and 12/6/2022 5:44:36 AM.
IsRegressionStdDev: Marked as regression because -79.32360504368846 (T) = (0 -38.70156206931072) / Math.Sqrt((0.0600271880374205 / (31)) + (0.176604933641807 / (19))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (19) - 2, .025) and -0.2774902003105302 = (30.29499722182073 - 38.70156206931072) / 30.29499722182073 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Matrix4x4.CreatePerspectiveFieldOfViewBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 59.82433701865941 > 51.15386678512785.
IsChangePoint: Marked as a change because one of 9/29/2022 8:01:59 PM, 12/1/2022 7:29:59 PM, 12/6/2022 5:44:36 AM falls between 11/27/2022 4:37:47 PM and 12/6/2022 5:44:36 AM.
IsRegressionStdDev: Marked as regression because -47.916905701150604 (T) = (0 -61.92219350297581) / Math.Sqrt((0.30119237792211584 / (31)) + (1.2266012173924523 / (19))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (19) - 2, .025) and -0.2672538707580032 = (48.86329008877856 - 61.92219350297581) / 48.86329008877856 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Matrix4x4.CreateScaleFromScalarWithCenterBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 61.04018148447524 > 34.79291111336848.
IsChangePoint: Marked as a change because one of 12/5/2022 3:14:07 PM, 12/6/2022 5:44:36 AM falls between 11/27/2022 4:37:47 PM and 12/6/2022 5:44:36 AM.
IsRegressionStdDev: Marked as regression because -259.2070065469351 (T) = (0 -61.14811376235012) / Math.Sqrt((0.0013583044208186747 / (48)) + (0.023298753214522308 / (2))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (48) + (2) - 2, .025) and -0.8452927073055777 = (33.137351879332 - 61.14811376235012) / 33.137351879332 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Matrix4x4.CreatePerspectiveOffCenterBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 39.46904267749687 > 33.314194843456775.
IsChangePoint: Marked as a change because one of 12/1/2022 7:29:59 PM, 12/6/2022 5:44:36 AM falls between 11/27/2022 4:37:47 PM and 12/6/2022 5:44:36 AM.
IsRegressionStdDev: Marked as regression because -59.151396848277585 (T) = (0 -39.76522244159256) / Math.Sqrt((0.043960989056271424 / (31)) + (0.31963797862347315 / (19))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (19) - 2, .025) and -0.2514134370082801 = (31.776246974507636 - 39.76522244159256) / 31.776246974507636 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Matrix4x4.CreateScaleFromVectorWithCenterBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 65.41711196077772 > 39.63991267273633.
IsChangePoint: Marked as a change because one of 12/5/2022 3:14:07 PM, 12/6/2022 5:44:36 AM falls between 11/27/2022 4:37:47 PM and 12/6/2022 5:44:36 AM.
IsRegressionStdDev: Marked as regression because -25.459822169364926 (T) = (0 -64.37259326890904) / Math.Sqrt((0.03014386824597277 / (48)) + (2.1820385953261234 / (2))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (48) + (2) - 2, .025) and -0.7042554697691138 = (37.77168060233951 - 64.37259326890904) / 37.77168060233951 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 776a87d9f0c9105c2729d1ab781d74bffa2a182b
Compare 9d7ffb5b0f9c56376460a0a45b90cb3ca90302ea
Diff Diff

Regressions in LinqBenchmarks

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Where01ForX - Duration of single invocation 449.49 ms 474.06 ms 1.05 0.09 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'LinqBenchmarks*'

Payloads

Baseline
Compare

Histogram

LinqBenchmarks.Where01ForX

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 474.0616430769231 > 470.4795391961539.
IsChangePoint: Marked as a change because one of 12/1/2022 7:29:59 PM, 12/6/2022 5:44:36 AM falls between 11/27/2022 4:37:47 PM and 12/6/2022 5:44:36 AM.
IsRegressionStdDev: Marked as regression because -11.414192163825925 (T) = (0 -483210714.0527574) / Math.Sqrt((15082204155246.945 / (31)) + (145205858199702.06 / (18))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (18) - 2, .025) and -0.07421141166345449 = (449828319.4594707 - 483210714.0527574) / 449828319.4594707 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
arch-x64 branch-refs/heads/main kind-micro_mono mono-aot os-linux perf-regression runtime-mono
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@naricc