Regressions in System.Memory.ReadOnlySpan.IndexOfString

[performanceautofiler]

Run Information

Name	Value
Architecture	arm64
OS	Windows 10.0.25094
Queue	AmpereWindows
Baseline	2b530d4abdab90801b9c3434d62e511d9f72e3e7
Compare	7a87fc1e18e53c03a17ae9105ad679d7906ae9ec
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Memory.ReadOnlySpan

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
IndexOfString - Duration of single invocation	27.69 ns	44.13 ns	1.59	0.69	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.ReadOnlySpan*'

Payloads

Baseline
Compare

Histogram

System.Memory.ReadOnlySpan.IndexOfString(input: "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAXAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", value: "x", comparisonType: OrdinalIgnoreCase)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 44.12726463340534 > 29.065976290113895.
IsChangePoint: Marked as a change because one of 5/1/2023 10:17:01 AM, 5/4/2023 2:44:24 AM falls between 4/25/2023 1:02:54 PM and 5/4/2023 2:44:24 AM.
IsRegressionStdDev: Marked as regression because -12.64424196287755 (T) = (0 -49.89955517156274) / Math.Sqrt((5.661681989159535 / (29)) + (22.471229788612526 / (8))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (8) - 2, .025) and -0.783145851023553 = (27.98400094020331 - 49.89955517156274) / 27.98400094020331 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[cincuranet]

Looks like #85437 could the culprit. cc @stephentoub

[cincuranet]

Same on x64:

• [Perf] Windows/x64: 5 Regressions on 4/25/2023 1:57:48 PM perf-autofiling-issues#17321
• [Perf] Windows/x64: 4 Regressions on 5/1/2023 3:42:23 PM perf-autofiling-issues#17224

[EgorBo]

I'd say it is expected - it is the worst case for the new algorithm

[stephentoub]

Presumably there are corresponding significant improvements in other benchmarks?

[cincuranet]

These are likely related and are improvements:

• [Perf] Linux/arm64: 79 Improvements on 4/26/2023 12:55:36 AM perf-autofiling-issues#17402
• [Perf] Windows/arm64: 106 Improvements on 4/26/2023 12:55:36 AM perf-autofiling-issues#17379
• [Perf] Windows/x64: 3 Improvements on 5/1/2023 3:42:23 PM perf-autofiling-issues#17341
• [Perf] Linux/x64: 79 Improvements on 4/25/2023 9:59:52 PM perf-autofiling-issues#17276
• [Perf] Windows/x86: 32 Improvements on 4/25/2023 8:51:41 AM perf-autofiling-issues#17254
• [Perf] Windows/x64: 67 Improvements on 4/25/2023 9:59:52 PM perf-autofiling-issues#17240

[ghost]

Tagging subscribers to this area: @dotnet/area-system-runtime
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Run Information

Name	Value
Architecture	arm64
OS	Windows 10.0.25094
Queue	AmpereWindows
Baseline	2b530d4abdab90801b9c3434d62e511d9f72e3e7
Compare	7a87fc1e18e53c03a17ae9105ad679d7906ae9ec
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Memory.ReadOnlySpan

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
IndexOfString - Duration of single invocation	27.69 ns	44.13 ns	1.59	0.69	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.ReadOnlySpan*'

Payloads

Baseline
Compare

Histogram

System.Memory.ReadOnlySpan.IndexOfString(input: "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAXAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", value: "x", comparisonType: OrdinalIgnoreCase)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 44.12726463340534 > 29.065976290113895.
IsChangePoint: Marked as a change because one of 5/1/2023 10:17:01 AM, 5/4/2023 2:44:24 AM falls between 4/25/2023 1:02:54 PM and 5/4/2023 2:44:24 AM.
IsRegressionStdDev: Marked as regression because -12.64424196287755 (T) = (0 -49.89955517156274) / Math.Sqrt((5.661681989159535 / (29)) + (22.471229788612526 / (8))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (8) - 2, .025) and -0.783145851023553 = (27.98400094020331 - 49.89955517156274) / 27.98400094020331 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author:	performanceautofiler[bot]
Assignees:	-
Labels:	arch-arm64, area-System.Runtime, os-windows, untriaged, runtime-coreclr, needs-area-label
Milestone:	-

[stephentoub]

This should also be fixed by #85781.