Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Regressions in System.Buffers.Tests.RentReturnArrayPoolTests<Byte> #70845

Closed
performanceautofiler bot opened this issue Jun 16, 2022 · 6 comments
Closed

[Perf] Regressions in System.Buffers.Tests.RentReturnArrayPoolTests<Byte> #70845

performanceautofiler bot opened this issue Jun 16, 2022 · 6 comments
Labels
arch-arm64 area-System.Threading runtime-coreclr specific to the CoreCLR runtime tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark
Milestone
7.0.0

Comments

@performanceautofiler
Copy link

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 3d74b00659fec817506e2888f87936518556e01c
Compare 0c3d5ad05754be529e470d7b0399f40a2bc8087d
Diff Diff

Regressions in System.Buffers.Tests.RentReturnArrayPoolTests<Byte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ProducerConsumer - Duration of single invocation 2.04 μs 2.22 μs 1.09 0.47 False
ProducerConsumer - Duration of single invocation 2.15 μs 2.70 μs 1.26 0.47 False

graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Buffers.Tests.RentReturnArrayPoolTests&lt;Byte&gt;*'

Payloads

Baseline
Compare

Histogram

System.Buffers.Tests.RentReturnArrayPoolTests<Byte>.ProducerConsumer(RentalSize: 4096, ManipulateArray: False, Async: False, UseSharedPool: False)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.222304787368421 > 2.1041162312624997.
IsChangePoint: Marked as a change because one of 6/7/2022 9:37:06 AM, 6/9/2022 7:18:42 PM, 6/15/2022 7:24:26 PM falls between 6/7/2022 5:42:36 AM and 6/15/2022 7:24:26 PM.
IsRegressionStdDev: Marked as regression because -7.940322094716294 (T) = (0 -2265.3246321699557) / Math.Sqrt((99975.27738342408 / (22)) + (12748.520185748039 / (24))) is less than -2.0153675744421933 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (24) - 2, .025) and -0.33283020571515504 = (1699.634824043062 - 2265.3246321699557) / 1699.634824043062 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Buffers.Tests.RentReturnArrayPoolTests&lt;Byte&gt;.ProducerConsumer(RentalSize: 4096, ManipulateArray: False, Async: False, UseSharedPool: True)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.701941663500001 > 2.3429245986250007.
IsChangePoint: Marked as a change because one of 6/7/2022 9:37:06 AM, 6/15/2022 7:24:26 PM falls between 6/7/2022 5:42:36 AM and 6/15/2022 7:24:26 PM.
IsRegressionStdDev: Marked as regression because -4.377755390345438 (T) = (0 -2646.81070288735) / Math.Sqrt((41494.895751718395 / (12)) + (30130.630066296122 / (34))) is less than -2.0153675744421933 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (34) - 2, .025) and -0.12235094938828425 = (2358.273679306765 - 2646.81070288735) / 2358.273679306765 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture arm64
OS ubuntu 20.04
Baseline 3d74b00659fec817506e2888f87936518556e01c
Compare 0c3d5ad05754be529e470d7b0399f40a2bc8087d
Diff Diff

Regressions in System.Buffers.Tests.RentReturnArrayPoolTests<Object>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ProducerConsumer - Duration of single invocation 2.06 μs 2.30 μs 1.12 0.45 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Buffers.Tests.RentReturnArrayPoolTests&lt;Object&gt;*'

Payloads

Baseline
Compare

Histogram

System.Buffers.Tests.RentReturnArrayPoolTests<Object>.ProducerConsumer(RentalSize: 4096, ManipulateArray: False, Async: False, UseSharedPool: False)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.302442016 > 2.0439055712250003.
IsChangePoint: Marked as a change because one of 6/7/2022 9:37:06 AM, 6/9/2022 7:18:42 PM, 6/15/2022 7:24:26 PM falls between 6/7/2022 5:42:36 AM and 6/15/2022 7:24:26 PM.
IsRegressionStdDev: Marked as regression because -7.992647096979788 (T) = (0 -2299.1375774301177) / Math.Sqrt((80590.29858937715 / (22)) + (17488.730522057453 / (24))) is less than -2.0153675744421933 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (24) - 2, .025) and -0.29934858212836735 = (1769.454024157297 - 2299.1375774301177) / 1769.454024157297 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler performanceautofiler bot added arm64 untriaged New issue has not been triaged by the area owner labels Jun 16, 2022
@DrewScoggins DrewScoggins transferred this issue from dotnet/perf-autofiling-issues Jun 16, 2022
@dotnet-issue-labeler
Copy link

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

@DrewScoggins
Copy link
Member

Could be related to #70478

@DrewScoggins DrewScoggins removed their assignment Jun 16, 2022
@DrewScoggins DrewScoggins changed the title [Perf] Changes at 6/10/2022 12:07:05 AM [Perf] Regressions in System.Buffers.Tests.RentReturnArrayPoolTests<Byte> Jun 16, 2022
@DrewScoggins DrewScoggins added tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark labels Jun 16, 2022
@ghost
Copy link

ghost commented Jun 22, 2022

Tagging subscribers to this area: @mangod9
See info in area-owners.md if you want to be subscribed.

Issue Details

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 3d74b00659fec817506e2888f87936518556e01c
Compare 0c3d5ad05754be529e470d7b0399f40a2bc8087d
Diff Diff

Regressions in System.Buffers.Tests.RentReturnArrayPoolTests<Byte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ProducerConsumer - Duration of single invocation 2.04 μs 2.22 μs 1.09 0.47 False
ProducerConsumer - Duration of single invocation 2.15 μs 2.70 μs 1.26 0.47 False

graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Buffers.Tests.RentReturnArrayPoolTests&lt;Byte&gt;*'

Payloads

Baseline
Compare

Histogram

System.Buffers.Tests.RentReturnArrayPoolTests<Byte>.ProducerConsumer(RentalSize: 4096, ManipulateArray: False, Async: False, UseSharedPool: False)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.222304787368421 > 2.1041162312624997.
IsChangePoint: Marked as a change because one of 6/7/2022 9:37:06 AM, 6/9/2022 7:18:42 PM, 6/15/2022 7:24:26 PM falls between 6/7/2022 5:42:36 AM and 6/15/2022 7:24:26 PM.
IsRegressionStdDev: Marked as regression because -7.940322094716294 (T) = (0 -2265.3246321699557) / Math.Sqrt((99975.27738342408 / (22)) + (12748.520185748039 / (24))) is less than -2.0153675744421933 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (24) - 2, .025) and -0.33283020571515504 = (1699.634824043062 - 2265.3246321699557) / 1699.634824043062 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Buffers.Tests.RentReturnArrayPoolTests&lt;Byte&gt;.ProducerConsumer(RentalSize: 4096, ManipulateArray: False, Async: False, UseSharedPool: True)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.701941663500001 > 2.3429245986250007.
IsChangePoint: Marked as a change because one of 6/7/2022 9:37:06 AM, 6/15/2022 7:24:26 PM falls between 6/7/2022 5:42:36 AM and 6/15/2022 7:24:26 PM.
IsRegressionStdDev: Marked as regression because -4.377755390345438 (T) = (0 -2646.81070288735) / Math.Sqrt((41494.895751718395 / (12)) + (30130.630066296122 / (34))) is less than -2.0153675744421933 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (34) - 2, .025) and -0.12235094938828425 = (2358.273679306765 - 2646.81070288735) / 2358.273679306765 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture arm64
OS ubuntu 20.04
Baseline 3d74b00659fec817506e2888f87936518556e01c
Compare 0c3d5ad05754be529e470d7b0399f40a2bc8087d
Diff Diff

Regressions in System.Buffers.Tests.RentReturnArrayPoolTests<Object>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ProducerConsumer - Duration of single invocation 2.06 μs 2.30 μs 1.12 0.45 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Buffers.Tests.RentReturnArrayPoolTests&lt;Object&gt;*'

Payloads

Baseline
Compare

Histogram

System.Buffers.Tests.RentReturnArrayPoolTests<Object>.ProducerConsumer(RentalSize: 4096, ManipulateArray: False, Async: False, UseSharedPool: False)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.302442016 > 2.0439055712250003.
IsChangePoint: Marked as a change because one of 6/7/2022 9:37:06 AM, 6/9/2022 7:18:42 PM, 6/15/2022 7:24:26 PM falls between 6/7/2022 5:42:36 AM and 6/15/2022 7:24:26 PM.
IsRegressionStdDev: Marked as regression because -7.992647096979788 (T) = (0 -2299.1375774301177) / Math.Sqrt((80590.29858937715 / (22)) + (17488.730522057453 / (24))) is less than -2.0153675744421933 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (24) - 2, .025) and -0.29934858212836735 = (1769.454024157297 - 2299.1375774301177) / 1769.454024157297 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author: performanceautofiler[bot]
Assignees: -
Labels:

area-System.Threading, tenet-performance, tenet-performance-benchmarks, untriaged, refs/heads/main, RunKind=micro, Regression, CoreClr, arm64, ubuntu 20.04
Milestone: -

@mangod9
Copy link
Member

mangod9 commented Jul 7, 2022

doesnt look related to #70478. @kouvel?

@mangod9 mangod9 removed the untriaged New issue has not been triaged by the area owner label Jul 7, 2022
@mangod9 mangod9 added this to the 7.0.0 milestone Jul 7, 2022
@mangod9
Copy link
Member

mangod9 commented Aug 4, 2022

I am closing this since looking at the chart there appears to be a dip and then a bump to a normal pattern, also the threadpool change should have nothing to do with this test.

@mangod9 mangod9 closed this as completed Aug 4, 2022
@kouvel
Copy link
Member

kouvel commented Aug 4, 2022

I've confirmed that #70478 is not the cause of the regression. On one of my arm64 machines, these tests have ~30-50% StdDev within a single run, so the change in value may not be significant. Looking at a narrower diff of changes that caused the relevant regression (example for the first test), it doesn't look like any other change would cause it either. It seems the regressions have recovered or are within reasonable error margin.

@ghost ghost locked as resolved and limited conversation to collaborators Sep 3, 2022
@jeffhandley jeffhandley added arch-arm64 runtime-coreclr specific to the CoreCLR runtime and removed arm64 labels Dec 28, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
arch-arm64 area-System.Threading runtime-coreclr specific to the CoreCLR runtime tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark
Projects
None yet
Milestone
7.0.0
Development

No branches or pull requests
5 participants
@jeffhandley @DrewScoggins @jeffschwMSFT @kouvel @mangod9