Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Linux/x64: 25 Regressions on 11/2/2022 4:25:40 PM #78812

Closed
performanceautofiler bot opened this issue Nov 8, 2022 · 7 comments
Closed

[Perf] Linux/x64: 25 Regressions on 11/2/2022 4:25:40 PM #78812

performanceautofiler bot opened this issue Nov 8, 2022 · 7 comments
Assignees
@tannergooding
Labels
arch-x64 area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI os-linux Linux OS (any supported distro) PGO
Milestone
8.0.0

Comments

@performanceautofiler
Copy link

Run Information

Architecture x64
OS ubuntu 18.04
Baseline bdd67af67dbf43c0f5e9151b98dc100b9555898a
Compare fbc20578253e0deb7267eea5a02b680c3cfe0e2a
Diff Diff

Regressions in System.Numerics.Tests.Perf_VectorOf<UInt16>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
InequalityOperatorBenchmark - Duration of single invocation 22.47 ns 101.87 ns 4.53 0.22 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;UInt16&gt;*'

Related Issues

Regressions

Improvements

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Numerics.Tests.Perf_VectorOf<UInt16>.InequalityOperatorBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 101.8723735640621 > 23.646291113151594.
IsChangePoint: Marked as a change because one of 11/2/2022 9:35:40 AM, 11/8/2022 6:56:37 AM falls between 10/30/2022 5:36:44 PM and 11/8/2022 6:56:37 AM.
IsRegressionStdDev: Marked as regression because -79.25538140347159 (T) = (0 -103.4858080440732) / Math.Sqrt((0.37671569915901404 / (27)) + (20.514024812935062 / (20))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (20) - 2, .025) and -3.5640094338646824 = (22.67431948676849 - 103.4858080440732) / 22.67431948676849 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline bdd67af67dbf43c0f5e9151b98dc100b9555898a
Compare 803688595257517f4b47e873d11152b0560c204a
Diff Diff

Regressions in System.Numerics.Tests.Perf_VectorOf<Double>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
BitwiseAndOperatorBenchmark - Duration of single invocation 17.05 ns 40.95 ns 2.40 0.26 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Double&gt;*'

Related Issues

Regressions

Improvements

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Numerics.Tests.Perf_VectorOf<Double>.BitwiseAndOperatorBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 40.95460802841224 > 17.89092744701803.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/8/2022 6:56:37 AM falls between 10/30/2022 5:36:44 PM and 11/8/2022 6:56:37 AM.
IsRegressionStdDev: Marked as regression because -61.7586337497908 (T) = (0 -40.92899037459878) / Math.Sqrt((3.550268126415335 / (27)) + (0.12886468606461166 / (15))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (15) - 2, .025) and -1.2975451308767367 = (17.814226943598882 - 40.92899037459878) / 17.814226943598882 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline bdd67af67dbf43c0f5e9151b98dc100b9555898a
Compare fbc20578253e0deb7267eea5a02b680c3cfe0e2a
Diff Diff

Regressions in System.Numerics.Tests.Perf_VectorOf<Int16>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
BitwiseAndOperatorBenchmark - Duration of single invocation 17.24 ns 46.23 ns 2.68 0.26 False
BitwiseOrOperatorBenchmark - Duration of single invocation 16.83 ns 41.12 ns 2.44 0.23 False
BitwiseAndBenchmark - Duration of single invocation 17.63 ns 41.01 ns 2.33 0.24 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Int16&gt;*'

Related Issues

Regressions

Improvements

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Numerics.Tests.Perf_VectorOf<Int16>.BitwiseAndOperatorBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 46.23171662618531 > 19.204274366632728.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/8/2022 6:56:37 AM falls between 10/30/2022 5:36:44 PM and 11/8/2022 6:56:37 AM.
IsRegressionStdDev: Marked as regression because -54.56970294733505 (T) = (0 -41.62094914874507) / Math.Sqrt((1.6947133397427672 / (27)) + (2.631937690725204 / (20))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (20) - 2, .025) and -1.3698157546097423 = (17.56294727460749 - 41.62094914874507) / 17.56294727460749 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int16&gt;.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 41.123642253570885 > 17.67390587361091.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/8/2022 6:56:37 AM falls between 10/30/2022 5:36:44 PM and 11/8/2022 6:56:37 AM.
IsRegressionStdDev: Marked as regression because -46.65548334193179 (T) = (0 -42.2698202917497) / Math.Sqrt((0.725433647671959 / (27)) + (5.146797230008012 / (20))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (20) - 2, .025) and -1.4296840421516723 = (17.39724983101775 - 42.2698202917497) / 17.39724983101775 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int16&gt;.BitwiseAndBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 41.00786437152644 > 18.106941145872657.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/8/2022 6:56:37 AM falls between 10/30/2022 5:36:44 PM and 11/8/2022 6:56:37 AM.
IsRegressionStdDev: Marked as regression because -47.971942304492586 (T) = (0 -41.89848386482377) / Math.Sqrt((0.9583090918739371 / (27)) + (3.369308689495923 / (15))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (15) - 2, .025) and -1.4035167613112252 = (17.432157969211033 - 41.89848386482377) / 17.432157969211033 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline bdd67af67dbf43c0f5e9151b98dc100b9555898a
Compare 803688595257517f4b47e873d11152b0560c204a
Diff Diff

Regressions in System.Numerics.Tests.Perf_VectorConvert

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Narrow_ulong - Duration of single invocation 79.78 μs 104.95 μs 1.32 0.20 False
Widen_float - Duration of single invocation 145.57 μs 205.33 μs 1.41 0.17 False
Convert_int_float - Duration of single invocation 73.89 μs 97.92 μs 1.33 0.24 False
Convert_double_ulong - Duration of single invocation 52.04 μs 81.86 μs 1.57 0.19 False
Convert_double_long - Duration of single invocation 49.22 μs 74.94 μs 1.52 0.22 False
Narrow_long - Duration of single invocation 79.00 μs 103.93 μs 1.32 0.23 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Numerics.Tests.Perf_VectorConvert*'

Related Issues

Regressions

Improvements

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Numerics.Tests.Perf_VectorConvert.Narrow_ulong

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 104.95298112416106 > 83.82619170245998.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/8/2022 6:56:37 AM falls between 10/30/2022 5:36:44 PM and 11/8/2022 6:56:37 AM.
IsRegressionStdDev: Marked as regression because -15.028197043380729 (T) = (0 -108498.67190792745) / Math.Sqrt((20640245.424279176 / (27)) + (41394914.23234916 / (15))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (15) - 2, .025) and -0.3513886502803742 = (80286.80119920876 - 108498.67190792745) / 80286.80119920876 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorConvert.Widen_float

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 205.33210570175436 > 152.89814630497685.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/8/2022 6:56:37 AM falls between 10/30/2022 5:36:44 PM and 11/8/2022 6:56:37 AM.
IsRegressionStdDev: Marked as regression because -46.03248274675961 (T) = (0 -206791.55442326108) / Math.Sqrt((24541186.451466978 / (27)) + (12172409.53414969 / (15))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (15) - 2, .025) and -0.41238491623099965 = (146413.0295125863 - 206791.55442326108) / 146413.0295125863 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorConvert.Convert_int_float

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 97.92457967355038 > 78.65790162056324.
IsChangePoint: Marked as a change because one of 9/28/2022 6:40:13 PM, 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/8/2022 6:56:37 AM falls between 10/30/2022 5:36:44 PM and 11/8/2022 6:56:37 AM.
IsRegressionStdDev: Marked as regression because -29.31764783983678 (T) = (0 -99345.42528445156) / Math.Sqrt((8551349.57005463 / (27)) + (9892849.651633168 / (20))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (20) - 2, .025) and -0.3620639825262765 = (72937.41451131502 - 99345.42528445156) / 72937.41451131502 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorConvert.Convert_double_ulong

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 81.86362716721594 > 54.66164615240883.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/8/2022 6:56:37 AM falls between 10/30/2022 5:36:44 PM and 11/8/2022 6:56:37 AM.
IsRegressionStdDev: Marked as regression because -40.13389904845121 (T) = (0 -79254.30966647006) / Math.Sqrt((2485774.038005878 / (27)) + (7364011.244383776 / (20))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (20) - 2, .025) and -0.5233509330800098 = (52026.29804166565 - 79254.30966647006) / 52026.29804166565 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorConvert.Convert_double_long

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 74.93954069199322 > 51.69727490201422.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/8/2022 6:56:37 AM falls between 10/30/2022 5:36:44 PM and 11/8/2022 6:56:37 AM.
IsRegressionStdDev: Marked as regression because -31.010454455633266 (T) = (0 -76429.69267691964) / Math.Sqrt((8255383.496955883 / (27)) + (8760225.391375972 / (20))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (20) - 2, .025) and -0.5382635855833126 = (49685.69326689051 - 76429.69267691964) / 49685.69326689051 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorConvert.Narrow_long

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 103.933983995585 > 82.88653661375476.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/8/2022 6:56:37 AM falls between 10/30/2022 5:36:44 PM and 11/8/2022 6:56:37 AM.
IsRegressionStdDev: Marked as regression because -29.327984135105677 (T) = (0 -104380.06520311843) / Math.Sqrt((16518238.505915344 / (27)) + (1416676.1425518903 / (15))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (15) - 2, .025) and -0.309111242962731 = (79733.53354363485 - 104380.06520311843) / 79733.53354363485 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline bdd67af67dbf43c0f5e9151b98dc100b9555898a
Compare 803688595257517f4b47e873d11152b0560c204a
Diff Diff

Regressions in System.Numerics.Tests.Perf_VectorOf<UInt32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
BitwiseOrOperatorBenchmark - Duration of single invocation 17.01 ns 41.21 ns 2.42 0.25 False
BitwiseAndOperatorBenchmark - Duration of single invocation 17.71 ns 41.00 ns 2.32 0.19 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;UInt32&gt;*'

Related Issues

Regressions

Improvements

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Numerics.Tests.Perf_VectorOf<UInt32>.BitwiseOrOperatorBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 41.209746789214364 > 17.66995864271164.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/8/2022 6:56:37 AM falls between 10/30/2022 5:36:44 PM and 11/8/2022 6:56:37 AM.
IsRegressionStdDev: Marked as regression because -89.54246276073859 (T) = (0 -41.62412908651522) / Math.Sqrt((0.26413727330486136 / (27)) + (0.9815957020861871 / (15))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (15) - 2, .025) and -1.4390725814118643 = (17.065555737755442 - 41.62412908651522) / 17.065555737755442 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt32&gt;.BitwiseAndOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 41.00427073607222 > 18.17271721873155.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/8/2022 6:56:37 AM falls between 10/30/2022 5:36:44 PM and 11/8/2022 6:56:37 AM.
IsRegressionStdDev: Marked as regression because -60.442393532133 (T) = (0 -41.659199935449834) / Math.Sqrt((0.6286674176296809 / (27)) + (2.087449203957097 / (15))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (15) - 2, .025) and -1.4083181882940397 = (17.298046469914183 - 41.659199935449834) / 17.298046469914183 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline bdd67af67dbf43c0f5e9151b98dc100b9555898a
Compare 803688595257517f4b47e873d11152b0560c204a
Diff Diff

Regressions in System.Numerics.Tests.Perf_VectorOf<UInt64>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
BitwiseAndBenchmark - Duration of single invocation 17.40 ns 41.48 ns 2.38 0.32 False
AndNotBenchmark - Duration of single invocation 47.23 ns 68.53 ns 1.45 0.21 False
BitwiseOrBenchmark - Duration of single invocation 16.99 ns 41.29 ns 2.43 0.27 False
BitwiseOrOperatorBenchmark - Duration of single invocation 16.68 ns 41.04 ns 2.46 0.25 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;UInt64&gt;*'

Related Issues

Regressions

Improvements

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Numerics.Tests.Perf_VectorOf<UInt64>.BitwiseAndBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 41.48398040340836 > 18.007254320146036.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/8/2022 6:56:37 AM falls between 10/30/2022 5:36:44 PM and 11/8/2022 6:56:37 AM.
IsRegressionStdDev: Marked as regression because -38.72305734344954 (T) = (0 -41.85410487326282) / Math.Sqrt((0.7372721120516357 / (27)) + (5.779947676987692 / (15))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (15) - 2, .025) and -1.4649547091155404 = (16.979664867059828 - 41.85410487326282) / 16.979664867059828 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt64&gt;.AndNotBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 68.53079839824615 > 49.59741285334486.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/8/2022 6:56:37 AM falls between 10/30/2022 5:36:44 PM and 11/8/2022 6:56:37 AM.
IsRegressionStdDev: Marked as regression because -26.743445228124298 (T) = (0 -69.569958025041) / Math.Sqrt((6.959244070546707 / (27)) + (6.191331620439273 / (15))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (15) - 2, .025) and -0.4593691978649538 = (47.6712528445998 - 69.569958025041) / 47.6712528445998 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt64&gt;.BitwiseOrBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 41.29102544643759 > 17.837997867879324.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/8/2022 6:56:37 AM falls between 10/30/2022 5:36:44 PM and 11/8/2022 6:56:37 AM.
IsRegressionStdDev: Marked as regression because -84.32194503404915 (T) = (0 -41.57919965089723) / Math.Sqrt((0.4316546353520266 / (27)) + (1.024167009304162 / (15))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (15) - 2, .025) and -1.4312679953394747 = (17.101857849731445 - 41.57919965089723) / 17.101857849731445 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt64&gt;.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 41.04113880156161 > 17.57886486632404.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/8/2022 6:56:37 AM falls between 10/30/2022 5:36:44 PM and 11/8/2022 6:56:37 AM.
IsRegressionStdDev: Marked as regression because -113.10015922039341 (T) = (0 -41.3385978615966) / Math.Sqrt((0.46826771432638165 / (27)) + (0.5724168770163124 / (20))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (20) - 2, .025) and -1.418765228745505 = (17.09078556707916 - 41.3385978615966) / 17.09078556707916 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline bdd67af67dbf43c0f5e9151b98dc100b9555898a
Compare 803688595257517f4b47e873d11152b0560c204a
Diff Diff

Regressions in System.Numerics.Tests.Perf_VectorOf<Single>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
InequalityOperatorBenchmark - Duration of single invocation 26.20 ns 66.18 ns 2.53 0.18 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 17.08 ns 41.70 ns 2.44 0.31 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Single&gt;*'

Related Issues

Regressions

Improvements

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Numerics.Tests.Perf_VectorOf<Single>.InequalityOperatorBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 66.18122802510553 > 25.716330206758997.
IsChangePoint: Marked as a change because one of 11/2/2022 9:35:40 AM, 11/8/2022 6:56:37 AM falls between 10/30/2022 5:36:44 PM and 11/8/2022 6:56:37 AM.
IsRegressionStdDev: Marked as regression because -72.04483808295696 (T) = (0 -66.92945269364074) / Math.Sqrt((0.8037325946287306 / (27)) + (4.77826178437499 / (15))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (15) - 2, .025) and -1.74192593849726 = (24.409650076224192 - 66.92945269364074) / 24.409650076224192 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Single&gt;.ExclusiveOrOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 41.70045337304935 > 17.94259740037124.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/8/2022 6:56:37 AM falls between 10/30/2022 5:36:44 PM and 11/8/2022 6:56:37 AM.
IsRegressionStdDev: Marked as regression because -115.66960144342421 (T) = (0 -41.31350116025097) / Math.Sqrt((0.506024790701155 / (27)) + (0.365550294310876 / (15))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (15) - 2, .025) and -1.3885310462048988 = (17.296614681183808 - 41.31350116025097) / 17.296614681183808 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler performanceautofiler bot added PGO refs/heads/main untriaged New issue has not been triaged by the area owner labels Nov 8, 2022
@performanceautofiler
Copy link
Author

Run Information

Architecture x64
OS ubuntu 18.04
Baseline bdd67af67dbf43c0f5e9151b98dc100b9555898a
Compare fbc20578253e0deb7267eea5a02b680c3cfe0e2a
Diff Diff

Regressions in System.Numerics.Tests.Perf_VectorOf<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
BitwiseOrBenchmark - Duration of single invocation 16.65 ns 41.07 ns 2.47 0.30 False
BitwiseAndBenchmark - Duration of single invocation 17.19 ns 41.54 ns 2.42 0.25 False
ConditionalSelectBenchmark - Duration of single invocation 81.32 ns 163.85 ns 2.01 0.25 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;*'

Related Issues

Regressions

Improvements

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Numerics.Tests.Perf_VectorOf<Int32>.BitwiseOrBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 41.07371789139586 > 17.82124160570579.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/8/2022 6:56:37 AM falls between 10/30/2022 5:36:44 PM and 11/8/2022 6:56:37 AM.
IsRegressionStdDev: Marked as regression because -55.96497802281049 (T) = (0 -41.2232290335979) / Math.Sqrt((4.400085385383967 / (27)) + (0.3600065367411615 / (20))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (20) - 2, .025) and -1.3670231697808852 = (17.415642381487093 - 41.2232290335979) / 17.415642381487093 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;.BitwiseAndBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 41.54379274314531 > 18.15630036838924.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/8/2022 6:56:37 AM falls between 10/30/2022 5:36:44 PM and 11/8/2022 6:56:37 AM.
IsRegressionStdDev: Marked as regression because -28.75870091857651 (T) = (0 -43.25361017662164) / Math.Sqrt((2.816639389897294 / (27)) + (13.656991826593853 / (20))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (20) - 2, .025) and -1.438451089778952 = (17.73814958106977 - 43.25361017662164) / 17.73814958106977 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;.ConditionalSelectBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 163.84726820439033 > 85.40522832527135.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/8/2022 6:56:37 AM falls between 10/30/2022 5:36:44 PM and 11/8/2022 6:56:37 AM.
IsRegressionStdDev: Marked as regression because -46.182119806481865 (T) = (0 -165.9423560599724) / Math.Sqrt((27.518118849257007 / (27)) + (44.43989665747486 / (20))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (20) - 2, .025) and -1.004150525595905 = (82.79934762416704 - 165.9423560599724) / 82.79934762416704 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline bdd67af67dbf43c0f5e9151b98dc100b9555898a
Compare fbc20578253e0deb7267eea5a02b680c3cfe0e2a
Diff Diff

Regressions in System.Numerics.Tests.Perf_VectorOf<Byte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
InequalityOperatorBenchmark - Duration of single invocation 20.77 ns 185.10 ns 8.91 0.19 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 17.09 ns 41.48 ns 2.43 0.26 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Byte&gt;*'

Related Issues

Regressions

Improvements

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Numerics.Tests.Perf_VectorOf<Byte>.InequalityOperatorBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 185.10435740942557 > 21.790545153099885.
IsChangePoint: Marked as a change because one of 11/2/2022 9:35:40 AM, 11/8/2022 6:56:37 AM falls between 10/30/2022 5:36:44 PM and 11/8/2022 6:56:37 AM.
IsRegressionStdDev: Marked as regression because -41.984557825259344 (T) = (0 -191.69914442255077) / Math.Sqrt((1.1284525527573832 / (27)) + (329.2070621993469 / (20))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (20) - 2, .025) and -8.065518524498977 = (21.145965771786383 - 191.69914442255077) / 21.145965771786383 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Byte&gt;.ExclusiveOrOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 41.48451343881469 > 18.694060454883797.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/8/2022 6:56:37 AM falls between 10/30/2022 5:36:44 PM and 11/8/2022 6:56:37 AM.
IsRegressionStdDev: Marked as regression because -139.89782237593263 (T) = (0 -41.206156058289324) / Math.Sqrt((0.6592430548118411 / (27)) + (0.09261551160118808 / (20))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (20) - 2, .025) and -1.3732106691657253 = (17.363041803943545 - 41.206156058289324) / 17.363041803943545 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline bdd67af67dbf43c0f5e9151b98dc100b9555898a
Compare fbc20578253e0deb7267eea5a02b680c3cfe0e2a
Diff Diff

Regressions in System.Numerics.Tests.Perf_VectorOf<SByte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
XorBenchmark - Duration of single invocation 17.26 ns 41.49 ns 2.40 0.30 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;*'

Related Issues

Regressions

Improvements

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Numerics.Tests.Perf_VectorOf<SByte>.XorBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 41.48998226869522 > 18.079516993444212.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/8/2022 6:56:37 AM falls between 10/30/2022 5:36:44 PM and 11/8/2022 6:56:37 AM.
IsRegressionStdDev: Marked as regression because -120.37718345776226 (T) = (0 -41.45055521174977) / Math.Sqrt((0.27533993513478067 / (27)) + (0.6040226759590084 / (20))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (20) - 2, .025) and -1.4021832841509119 = (17.255367433963766 - 41.45055521174977) / 17.255367433963766 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@kotlarmilos
Copy link
Member

kotlarmilos commented Nov 9, 2022 •
edited
Loading

regressions in 13d4e6d

@kotlarmilos kotlarmilos removed the untriaged New issue has not been triaged by the area owner label Nov 9, 2022
@kotlarmilos kotlarmilos mentioned this issue Nov 9, 2022
Closed
@kotlarmilos
Copy link
Member

Some regressions are fixed by #78182 and #78177

@kotlarmilos
Copy link
Member

Hey @tannergooding still some regressions in BitwiseOrBenchmark. Do you have any idea what could be an issue?

@dotnet-issue-labeler
Copy link

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

@ghost ghost added the untriaged New issue has not been triaged by the area owner label Nov 24, 2022
@kotlarmilos kotlarmilos transferred this issue from dotnet/perf-autofiling-issues Nov 24, 2022
@kotlarmilos kotlarmilos removed the untriaged New issue has not been triaged by the area owner label Nov 24, 2022
@jeffschwMSFT jeffschwMSFT added the area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI label Nov 25, 2022
@ghost
Copy link

ghost commented Nov 25, 2022

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

Issue Details

Run Information

Architecture x64
OS ubuntu 18.04
Baseline bdd67af67dbf43c0f5e9151b98dc100b9555898a
Compare fbc20578253e0deb7267eea5a02b680c3cfe0e2a
Diff Diff

Regressions in System.Numerics.Tests.Perf_VectorOf<UInt16>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
InequalityOperatorBenchmark - Duration of single invocation 22.47 ns 101.87 ns 4.53 0.22 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;UInt16&gt;*'

Related Issues

Regressions

Improvements

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Numerics.Tests.Perf_VectorOf<UInt16>.InequalityOperatorBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 101.8723735640621 > 23.646291113151594.
IsChangePoint: Marked as a change because one of 11/2/2022 9:35:40 AM, 11/8/2022 6:56:37 AM falls between 10/30/2022 5:36:44 PM and 11/8/2022 6:56:37 AM.
IsRegressionStdDev: Marked as regression because -79.25538140347159 (T) = (0 -103.4858080440732) / Math.Sqrt((0.37671569915901404 / (27)) + (20.514024812935062 / (20))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (20) - 2, .025) and -3.5640094338646824 = (22.67431948676849 - 103.4858080440732) / 22.67431948676849 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline bdd67af67dbf43c0f5e9151b98dc100b9555898a
Compare 803688595257517f4b47e873d11152b0560c204a
Diff Diff

Regressions in System.Numerics.Tests.Perf_VectorOf<Double>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
BitwiseAndOperatorBenchmark - Duration of single invocation 17.05 ns 40.95 ns 2.40 0.26 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Double&gt;*'

Related Issues

Regressions

Improvements

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Numerics.Tests.Perf_VectorOf<Double>.BitwiseAndOperatorBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 40.95460802841224 > 17.89092744701803.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/8/2022 6:56:37 AM falls between 10/30/2022 5:36:44 PM and 11/8/2022 6:56:37 AM.
IsRegressionStdDev: Marked as regression because -61.7586337497908 (T) = (0 -40.92899037459878) / Math.Sqrt((3.550268126415335 / (27)) + (0.12886468606461166 / (15))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (15) - 2, .025) and -1.2975451308767367 = (17.814226943598882 - 40.92899037459878) / 17.814226943598882 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline bdd67af67dbf43c0f5e9151b98dc100b9555898a
Compare fbc20578253e0deb7267eea5a02b680c3cfe0e2a
Diff Diff

Regressions in System.Numerics.Tests.Perf_VectorOf<Int16>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
BitwiseAndOperatorBenchmark - Duration of single invocation 17.24 ns 46.23 ns 2.68 0.26 False
BitwiseOrOperatorBenchmark - Duration of single invocation 16.83 ns 41.12 ns 2.44 0.23 False
BitwiseAndBenchmark - Duration of single invocation 17.63 ns 41.01 ns 2.33 0.24 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Int16&gt;*'

Related Issues

Regressions

Improvements

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Numerics.Tests.Perf_VectorOf<Int16>.BitwiseAndOperatorBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 46.23171662618531 > 19.204274366632728.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/8/2022 6:56:37 AM falls between 10/30/2022 5:36:44 PM and 11/8/2022 6:56:37 AM.
IsRegressionStdDev: Marked as regression because -54.56970294733505 (T) = (0 -41.62094914874507) / Math.Sqrt((1.6947133397427672 / (27)) + (2.631937690725204 / (20))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (20) - 2, .025) and -1.3698157546097423 = (17.56294727460749 - 41.62094914874507) / 17.56294727460749 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int16&gt;.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 41.123642253570885 > 17.67390587361091.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/8/2022 6:56:37 AM falls between 10/30/2022 5:36:44 PM and 11/8/2022 6:56:37 AM.
IsRegressionStdDev: Marked as regression because -46.65548334193179 (T) = (0 -42.2698202917497) / Math.Sqrt((0.725433647671959 / (27)) + (5.146797230008012 / (20))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (20) - 2, .025) and -1.4296840421516723 = (17.39724983101775 - 42.2698202917497) / 17.39724983101775 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int16&gt;.BitwiseAndBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 41.00786437152644 > 18.106941145872657.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/8/2022 6:56:37 AM falls between 10/30/2022 5:36:44 PM and 11/8/2022 6:56:37 AM.
IsRegressionStdDev: Marked as regression because -47.971942304492586 (T) = (0 -41.89848386482377) / Math.Sqrt((0.9583090918739371 / (27)) + (3.369308689495923 / (15))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (15) - 2, .025) and -1.4035167613112252 = (17.432157969211033 - 41.89848386482377) / 17.432157969211033 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline bdd67af67dbf43c0f5e9151b98dc100b9555898a
Compare 803688595257517f4b47e873d11152b0560c204a
Diff Diff

Regressions in System.Numerics.Tests.Perf_VectorConvert

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Narrow_ulong - Duration of single invocation 79.78 μs 104.95 μs 1.32 0.20 False
Widen_float - Duration of single invocation 145.57 μs 205.33 μs 1.41 0.17 False
Convert_int_float - Duration of single invocation 73.89 μs 97.92 μs 1.33 0.24 False
Convert_double_ulong - Duration of single invocation 52.04 μs 81.86 μs 1.57 0.19 False
Convert_double_long - Duration of single invocation 49.22 μs 74.94 μs 1.52 0.22 False
Narrow_long - Duration of single invocation 79.00 μs 103.93 μs 1.32 0.23 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Numerics.Tests.Perf_VectorConvert*'

Related Issues

Regressions

Improvements

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Numerics.Tests.Perf_VectorConvert.Narrow_ulong

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 104.95298112416106 > 83.82619170245998.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/8/2022 6:56:37 AM falls between 10/30/2022 5:36:44 PM and 11/8/2022 6:56:37 AM.
IsRegressionStdDev: Marked as regression because -15.028197043380729 (T) = (0 -108498.67190792745) / Math.Sqrt((20640245.424279176 / (27)) + (41394914.23234916 / (15))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (15) - 2, .025) and -0.3513886502803742 = (80286.80119920876 - 108498.67190792745) / 80286.80119920876 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorConvert.Widen_float

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 205.33210570175436 > 152.89814630497685.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/8/2022 6:56:37 AM falls between 10/30/2022 5:36:44 PM and 11/8/2022 6:56:37 AM.
IsRegressionStdDev: Marked as regression because -46.03248274675961 (T) = (0 -206791.55442326108) / Math.Sqrt((24541186.451466978 / (27)) + (12172409.53414969 / (15))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (15) - 2, .025) and -0.41238491623099965 = (146413.0295125863 - 206791.55442326108) / 146413.0295125863 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorConvert.Convert_int_float

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 97.92457967355038 > 78.65790162056324.
IsChangePoint: Marked as a change because one of 9/28/2022 6:40:13 PM, 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/8/2022 6:56:37 AM falls between 10/30/2022 5:36:44 PM and 11/8/2022 6:56:37 AM.
IsRegressionStdDev: Marked as regression because -29.31764783983678 (T) = (0 -99345.42528445156) / Math.Sqrt((8551349.57005463 / (27)) + (9892849.651633168 / (20))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (20) - 2, .025) and -0.3620639825262765 = (72937.41451131502 - 99345.42528445156) / 72937.41451131502 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorConvert.Convert_double_ulong

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 81.86362716721594 > 54.66164615240883.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/8/2022 6:56:37 AM falls between 10/30/2022 5:36:44 PM and 11/8/2022 6:56:37 AM.
IsRegressionStdDev: Marked as regression because -40.13389904845121 (T) = (0 -79254.30966647006) / Math.Sqrt((2485774.038005878 / (27)) + (7364011.244383776 / (20))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (20) - 2, .025) and -0.5233509330800098 = (52026.29804166565 - 79254.30966647006) / 52026.29804166565 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorConvert.Convert_double_long

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 74.93954069199322 > 51.69727490201422.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/8/2022 6:56:37 AM falls between 10/30/2022 5:36:44 PM and 11/8/2022 6:56:37 AM.
IsRegressionStdDev: Marked as regression because -31.010454455633266 (T) = (0 -76429.69267691964) / Math.Sqrt((8255383.496955883 / (27)) + (8760225.391375972 / (20))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (20) - 2, .025) and -0.5382635855833126 = (49685.69326689051 - 76429.69267691964) / 49685.69326689051 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorConvert.Narrow_long

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 103.933983995585 > 82.88653661375476.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/8/2022 6:56:37 AM falls between 10/30/2022 5:36:44 PM and 11/8/2022 6:56:37 AM.
IsRegressionStdDev: Marked as regression because -29.327984135105677 (T) = (0 -104380.06520311843) / Math.Sqrt((16518238.505915344 / (27)) + (1416676.1425518903 / (15))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (15) - 2, .025) and -0.309111242962731 = (79733.53354363485 - 104380.06520311843) / 79733.53354363485 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline bdd67af67dbf43c0f5e9151b98dc100b9555898a
Compare 803688595257517f4b47e873d11152b0560c204a
Diff Diff

Regressions in System.Numerics.Tests.Perf_VectorOf<UInt32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
BitwiseOrOperatorBenchmark - Duration of single invocation 17.01 ns 41.21 ns 2.42 0.25 False
BitwiseAndOperatorBenchmark - Duration of single invocation 17.71 ns 41.00 ns 2.32 0.19 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;UInt32&gt;*'

Related Issues

Regressions

Improvements

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Numerics.Tests.Perf_VectorOf<UInt32>.BitwiseOrOperatorBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 41.209746789214364 > 17.66995864271164.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/8/2022 6:56:37 AM falls between 10/30/2022 5:36:44 PM and 11/8/2022 6:56:37 AM.
IsRegressionStdDev: Marked as regression because -89.54246276073859 (T) = (0 -41.62412908651522) / Math.Sqrt((0.26413727330486136 / (27)) + (0.9815957020861871 / (15))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (15) - 2, .025) and -1.4390725814118643 = (17.065555737755442 - 41.62412908651522) / 17.065555737755442 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt32&gt;.BitwiseAndOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 41.00427073607222 > 18.17271721873155.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/8/2022 6:56:37 AM falls between 10/30/2022 5:36:44 PM and 11/8/2022 6:56:37 AM.
IsRegressionStdDev: Marked as regression because -60.442393532133 (T) = (0 -41.659199935449834) / Math.Sqrt((0.6286674176296809 / (27)) + (2.087449203957097 / (15))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (15) - 2, .025) and -1.4083181882940397 = (17.298046469914183 - 41.659199935449834) / 17.298046469914183 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline bdd67af67dbf43c0f5e9151b98dc100b9555898a
Compare 803688595257517f4b47e873d11152b0560c204a
Diff Diff

Regressions in System.Numerics.Tests.Perf_VectorOf<UInt64>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
BitwiseAndBenchmark - Duration of single invocation 17.40 ns 41.48 ns 2.38 0.32 False
AndNotBenchmark - Duration of single invocation 47.23 ns 68.53 ns 1.45 0.21 False
BitwiseOrBenchmark - Duration of single invocation 16.99 ns 41.29 ns 2.43 0.27 False
BitwiseOrOperatorBenchmark - Duration of single invocation 16.68 ns 41.04 ns 2.46 0.25 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;UInt64&gt;*'

Related Issues

Regressions

Improvements

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Numerics.Tests.Perf_VectorOf<UInt64>.BitwiseAndBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 41.48398040340836 > 18.007254320146036.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/8/2022 6:56:37 AM falls between 10/30/2022 5:36:44 PM and 11/8/2022 6:56:37 AM.
IsRegressionStdDev: Marked as regression because -38.72305734344954 (T) = (0 -41.85410487326282) / Math.Sqrt((0.7372721120516357 / (27)) + (5.779947676987692 / (15))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (15) - 2, .025) and -1.4649547091155404 = (16.979664867059828 - 41.85410487326282) / 16.979664867059828 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt64&gt;.AndNotBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 68.53079839824615 > 49.59741285334486.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/8/2022 6:56:37 AM falls between 10/30/2022 5:36:44 PM and 11/8/2022 6:56:37 AM.
IsRegressionStdDev: Marked as regression because -26.743445228124298 (T) = (0 -69.569958025041) / Math.Sqrt((6.959244070546707 / (27)) + (6.191331620439273 / (15))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (15) - 2, .025) and -0.4593691978649538 = (47.6712528445998 - 69.569958025041) / 47.6712528445998 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt64&gt;.BitwiseOrBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 41.29102544643759 > 17.837997867879324.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/8/2022 6:56:37 AM falls between 10/30/2022 5:36:44 PM and 11/8/2022 6:56:37 AM.
IsRegressionStdDev: Marked as regression because -84.32194503404915 (T) = (0 -41.57919965089723) / Math.Sqrt((0.4316546353520266 / (27)) + (1.024167009304162 / (15))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (15) - 2, .025) and -1.4312679953394747 = (17.101857849731445 - 41.57919965089723) / 17.101857849731445 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt64&gt;.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 41.04113880156161 > 17.57886486632404.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/8/2022 6:56:37 AM falls between 10/30/2022 5:36:44 PM and 11/8/2022 6:56:37 AM.
IsRegressionStdDev: Marked as regression because -113.10015922039341 (T) = (0 -41.3385978615966) / Math.Sqrt((0.46826771432638165 / (27)) + (0.5724168770163124 / (20))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (20) - 2, .025) and -1.418765228745505 = (17.09078556707916 - 41.3385978615966) / 17.09078556707916 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline bdd67af67dbf43c0f5e9151b98dc100b9555898a
Compare 803688595257517f4b47e873d11152b0560c204a
Diff Diff

Regressions in System.Numerics.Tests.Perf_VectorOf<Single>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
InequalityOperatorBenchmark - Duration of single invocation 26.20 ns 66.18 ns 2.53 0.18 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 17.08 ns 41.70 ns 2.44 0.31 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Single&gt;*'

Related Issues

Regressions

Improvements

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Numerics.Tests.Perf_VectorOf<Single>.InequalityOperatorBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 66.18122802510553 > 25.716330206758997.
IsChangePoint: Marked as a change because one of 11/2/2022 9:35:40 AM, 11/8/2022 6:56:37 AM falls between 10/30/2022 5:36:44 PM and 11/8/2022 6:56:37 AM.
IsRegressionStdDev: Marked as regression because -72.04483808295696 (T) = (0 -66.92945269364074) / Math.Sqrt((0.8037325946287306 / (27)) + (4.77826178437499 / (15))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (15) - 2, .025) and -1.74192593849726 = (24.409650076224192 - 66.92945269364074) / 24.409650076224192 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Single&gt;.ExclusiveOrOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 41.70045337304935 > 17.94259740037124.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/8/2022 6:56:37 AM falls between 10/30/2022 5:36:44 PM and 11/8/2022 6:56:37 AM.
IsRegressionStdDev: Marked as regression because -115.66960144342421 (T) = (0 -41.31350116025097) / Math.Sqrt((0.506024790701155 / (27)) + (0.365550294310876 / (15))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (15) - 2, .025) and -1.3885310462048988 = (17.296614681183808 - 41.31350116025097) / 17.296614681183808 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author: performanceautofiler[bot]
Assignees: tannergooding
Labels:

area-CodeGen-coreclr, refs/heads/main, ubuntu 18.04, PGO, Regression, x64
Milestone: -

@JulieLeeMSFT JulieLeeMSFT added this to the 8.0.0 milestone Nov 26, 2022
@kotlarmilos
Copy link
Member

Fixed by b77aa8a

@jeffhandley jeffhandley added os-linux Linux OS (any supported distro) arch-x64 and removed refs/heads/main labels Dec 28, 2022
@ghost ghost locked as resolved and limited conversation to collaborators Jan 27, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@tannergooding tannergooding

Labels
arch-x64 area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI os-linux Linux OS (any supported distro) PGO
Projects
None yet
Milestone
8.0.0
Development

No branches or pull requests
5 participants
@jeffhandley @jeffschwMSFT @tannergooding @kotlarmilos @JulieLeeMSFT