Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Common #61972

Closed
performanceautofiler bot opened this issue Nov 23, 2021 · 8 comments · Fixed by #67184
Closed

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Common #61972

performanceautofiler bot opened this issue Nov 23, 2021 · 8 comments · Fixed by #67184
Assignees
@joperezr
Labels
area-System.Text.RegularExpressions Priority:0 Work that we can't release without tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark
Milestone
7.0.0

Comments

@performanceautofiler
Copy link

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 44d28bf5b6c8b448ac4f19638b90cdbdba6be5bc
Compare 6ff57f1935d1cb3396674bec26268a301e5c0628
Diff Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Common

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Email_IsNotMatch - Duration of single invocation 334.65 ns 587.91 ns 1.76 0.03 True

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Common*'

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Common.Email_IsNotMatch(Options: IgnoreCase, Compiled)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 587.9140948889302 > 353.26904849015745.
IsChangePoint: Marked as a change because one of 11/17/2021 2:42:53 PM, 11/23/2021 3:51:55 AM falls between 11/14/2021 12:07:47 PM and 11/23/2021 3:51:55 AM.
IsRegressionStdDev: Marked as regression because -70.5828741771422 (T) = (0 -583.6719188293049) / Math.Sqrt((20.548479861772265 / (33)) + (225.46833403751063 / (20))) is less than -2.007583770314729 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (20) - 2, .025) and -0.7155392558514462 = (340.2265012814413 - 583.6719188293049) / 340.2265012814413 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS Windows 10.0.18362
Baseline 44d28bf5b6c8b448ac4f19638b90cdbdba6be5bc
Compare 9962c10813de695782baa20f1bf0bb2e9810971d
Diff Diff

Regressions in PerfLabTests.CastingPerf

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IFooFooIsIFoo - Duration of single invocation 218.29 μs 280.65 μs 1.29 0.38 False
ObjInt - Duration of single invocation 249.46 μs 280.64 μs 1.12 0.37 False

graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'PerfLabTests.CastingPerf*'

Payloads

Baseline
Compare

Histogram

PerfLabTests.CastingPerf.IFooFooIsIFoo

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 280.6496565934066 > 261.9432088293651.
IsChangePoint: Marked as a change because one of 10/16/2021 4:18:44 PM, 11/11/2021 11:33:25 PM, 11/17/2021 2:42:53 PM, 11/23/2021 3:51:55 AM falls between 11/14/2021 12:07:47 PM and 11/23/2021 3:51:55 AM.
IsRegressionStdDev: Marked as regression because -10.231182091581285 (T) = (0 -308430.1068510492) / Math.Sqrt((331433289.3165045 / (33)) + (444218998.4880154 / (26))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (26) - 2, .025) and -0.20886285352633047 = (255140.69354628507 - 308430.1068510492) / 255140.69354628507 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### PerfLabTests.CastingPerf.ObjInt

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 280.6436727335165 > 261.94455902777776.
IsChangePoint: Marked as a change because one of 9/22/2021 5:04:00 PM, 10/29/2021 10:34:07 PM, 11/11/2021 11:33:25 PM, 11/17/2021 2:42:53 PM, 11/23/2021 3:51:55 AM falls between 11/14/2021 12:07:47 PM and 11/23/2021 3:51:55 AM.
IsRegressionStdDev: Marked as regression because -9.082524798950823 (T) = (0 -308658.5429815293) / Math.Sqrt((441956257.76034564 / (33)) + (461405557.2852367 / (26))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (26) - 2, .025) and -0.19646245517031016 = (257975.95373570948 - 308658.5429815293) / 257975.95373570948 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@AndyAyersMS
Copy link
Member

Suspect the Regex is from #61490.

The others are noise.

@AndyAyersMS AndyAyersMS transferred this issue from dotnet/perf-autofiling-issues Nov 23, 2021
@dotnet-issue-labeler
Copy link

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

@dotnet-issue-labeler dotnet-issue-labeler bot added the untriaged New issue has not been triaged by the area owner label Nov 23, 2021
@AndyAyersMS AndyAyersMS added arch-x64 os-windows tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark labels Nov 23, 2021
@AndyAyersMS AndyAyersMS changed the title [Perf] Changes at 11/17/2021 8:43:26 PM Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Common Nov 23, 2021
@AndyAyersMS
Copy link
Member

cc @stephentoub

@stephentoub
Copy link
Member

stephentoub commented Nov 23, 2021 •
edited
Loading

This is the same issue as #61786 (comment). The pattern contains ([a-zA-Z0-9_\-\.]+)@. Previously that loop would have been made atomic, but because the @ is now being changed to be case-sensitive, the auto-atomicity optimization bails out immediately upon seeing that the set differs in case-sensitivity from the node following it. The solution here is similarly #61048. That will fix this and more.
cc: @joperezr

@AndyAyersMS
Copy link
Member

Also seen on x86: dotnet/perf-autofiling-issues#2514

@stephentoub
Copy link
Member

Also seen on x86

I'd expect this to regress similarly regardless of architecture.

@ghost
Copy link

ghost commented Nov 25, 2021

Tagging subscribers to this area: @eerhardt, @dotnet/area-system-text-regularexpressions
See info in area-owners.md if you want to be subscribed.

Issue Details

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 44d28bf5b6c8b448ac4f19638b90cdbdba6be5bc
Compare 6ff57f1935d1cb3396674bec26268a301e5c0628
Diff Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Common

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Email_IsNotMatch - Duration of single invocation 334.65 ns 587.91 ns 1.76 0.03 True

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Common*'

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Common.Email_IsNotMatch(Options: IgnoreCase, Compiled)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 587.9140948889302 > 353.26904849015745.
IsChangePoint: Marked as a change because one of 11/17/2021 2:42:53 PM, 11/23/2021 3:51:55 AM falls between 11/14/2021 12:07:47 PM and 11/23/2021 3:51:55 AM.
IsRegressionStdDev: Marked as regression because -70.5828741771422 (T) = (0 -583.6719188293049) / Math.Sqrt((20.548479861772265 / (33)) + (225.46833403751063 / (20))) is less than -2.007583770314729 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (20) - 2, .025) and -0.7155392558514462 = (340.2265012814413 - 583.6719188293049) / 340.2265012814413 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS Windows 10.0.18362
Baseline 44d28bf5b6c8b448ac4f19638b90cdbdba6be5bc
Compare 9962c10813de695782baa20f1bf0bb2e9810971d
Diff Diff

Regressions in PerfLabTests.CastingPerf

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IFooFooIsIFoo - Duration of single invocation 218.29 μs 280.65 μs 1.29 0.38 False
ObjInt - Duration of single invocation 249.46 μs 280.64 μs 1.12 0.37 False

graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'PerfLabTests.CastingPerf*'

Payloads

Baseline
Compare

Histogram

PerfLabTests.CastingPerf.IFooFooIsIFoo

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 280.6496565934066 > 261.9432088293651.
IsChangePoint: Marked as a change because one of 10/16/2021 4:18:44 PM, 11/11/2021 11:33:25 PM, 11/17/2021 2:42:53 PM, 11/23/2021 3:51:55 AM falls between 11/14/2021 12:07:47 PM and 11/23/2021 3:51:55 AM.
IsRegressionStdDev: Marked as regression because -10.231182091581285 (T) = (0 -308430.1068510492) / Math.Sqrt((331433289.3165045 / (33)) + (444218998.4880154 / (26))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (26) - 2, .025) and -0.20886285352633047 = (255140.69354628507 - 308430.1068510492) / 255140.69354628507 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### PerfLabTests.CastingPerf.ObjInt

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 280.6436727335165 > 261.94455902777776.
IsChangePoint: Marked as a change because one of 9/22/2021 5:04:00 PM, 10/29/2021 10:34:07 PM, 11/11/2021 11:33:25 PM, 11/17/2021 2:42:53 PM, 11/23/2021 3:51:55 AM falls between 11/14/2021 12:07:47 PM and 11/23/2021 3:51:55 AM.
IsRegressionStdDev: Marked as regression because -9.082524798950823 (T) = (0 -308658.5429815293) / Math.Sqrt((441956257.76034564 / (33)) + (461405557.2852367 / (26))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (26) - 2, .025) and -0.19646245517031016 = (257975.95373570948 - 308658.5429815293) / 257975.95373570948 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author: performanceautofiler[bot]
Assignees: -
Labels:

area-System.Text.RegularExpressions, os-windows, tenet-performance, tenet-performance-benchmarks, arch-x64, untriaged
Milestone: -

@stephentoub stephentoub removed os-windows arch-x64 untriaged New issue has not been triaged by the area owner labels Dec 10, 2021
@joperezr joperezr mentioned this issue Dec 14, 2021
Closed
48 tasks
This was referenced Dec 20, 2021
Closed
Closed
@naricc
Copy link
Contributor

naricc commented Dec 20, 2021

FYI, this regression is also effecting mono similarly.

@naricc naricc closed this as completed Dec 20, 2021
@naricc naricc reopened this Dec 20, 2021
This was referenced Dec 20, 2021
Closed
Closed
@ericstj ericstj added the Priority:0 Work that we can't release without label Jan 10, 2022
@joperezr joperezr self-assigned this Jan 14, 2022
@joperezr joperezr added this to the 7.0.0 milestone Jan 14, 2022
@stephentoub stephentoub mentioned this issue Mar 26, 2022
Merged
@ghost ghost added the in-pr There is an active PR which will close this issue when it is merged label Mar 29, 2022
@ghost ghost removed the in-pr There is an active PR which will close this issue when it is merged label Apr 6, 2022
@ghost ghost locked as resolved and limited conversation to collaborators May 6, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@joperezr joperezr

Labels
area-System.Text.RegularExpressions Priority:0 Work that we can't release without tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark
Projects
None yet
Milestone
7.0.0
Development

Successfully merging a pull request may close this issue.

Changing the logic for how we deal with RegexOptions.IgnoreCase matching. joperezr/runtime
6 participants
@stephentoub @ericstj @jeffschwMSFT @AndyAyersMS @joperezr @naricc