Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Mono regression in IndexOf-based search using Contains method #21818

Closed
performanceautofiler bot opened this issue Sep 19, 2023 · 20 comments
Closed

[Perf] Mono regression in IndexOf-based search using Contains method #21818

performanceautofiler bot opened this issue Sep 19, 2023 · 20 comments
Assignees
@kotlarmilos @MihaZupan @fanyang-mono
Labels
arch-x64 branch-refs/heads/main kind-micro_mono LinkedFromMaster Is this issue linked from the master perf issue mono-aot os-linux perf-regression PGO Applied if there were any profile guided optimization updates in the observed interval. runtime-mono

Comments

@performanceautofiler
Copy link

performanceautofiler bot commented Sep 19, 2023 •
edited
Loading

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 9d08b24d743d0c57203a55c3d0c6dd1bc472e57e
Compare f6c592995a4f8526508da33761230a3850d942ff
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.Memory.ReadOnlySpan

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
101.71 ns 756.72 ns 7.44 0.01 True
798.38 ns 916.26 ns 1.15 0.01 True
61.08 ns 152.32 ns 2.49 0.01 True
129.35 ns 633.10 ns 4.89 0.01 True
659.54 ns 781.55 ns 1.18 0.32 True
84.07 ns 131.99 ns 1.57 0.01 True

graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Memory.ReadOnlySpan*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Memory.ReadOnlySpan* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Memory.ReadOnlySpan*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Memory.ReadOnlySpan* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

System.Memory.ReadOnlySpan.IndexOfString(input: "string1", value: "string2", comparisonType: InvariantCulture)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 756.718505146577 > 106.34061876957904.
IsChangePoint: Marked as a change because one of 9/11/2023 8:56:48 PM, 9/18/2023 8:11:22 PM falls between 9/9/2023 9:29:58 PM and 9/18/2023 8:11:22 PM.
IsRegressionStdDev: Marked as regression because -811.6645095725975 (T) = (0 -761.7053348947214) / Math.Sqrt((0.7834027141297717 / (11)) + (14.175208381211368 / (24))) is less than -2.034515297446192 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (11) + (24) - 2, .025) and -6.513264694472396 = (101.38140553667937 - 761.7053348947214) / 101.38140553667937 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Memory.ReadOnlySpan.IndexOfString(input: "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAXAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", value: "x", comparisonType: InvariantCultureIgnoreCase)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 916.2617702704026 > 834.0282095096256.
IsChangePoint: Marked as a change because one of 9/11/2023 8:56:48 PM, 9/18/2023 8:11:22 PM falls between 9/9/2023 9:29:58 PM and 9/18/2023 8:11:22 PM.
IsRegressionStdDev: Marked as regression because -75.00358844773847 (T) = (0 -914.6770370873656) / Math.Sqrt((25.706585312583464 / (11)) + (5.045886120712956 / (23))) is less than -2.03693334345674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (11) + (23) - 2, .025) and -0.15088906704142913 = (794.7569086208372 - 914.6770370873656) / 794.7569086208372 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Memory.ReadOnlySpan.IndexOfString(input: "ABCDE", value: "c", comparisonType: InvariantCultureIgnoreCase)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 152.3165715164346 > 64.13606464812156.
IsChangePoint: Marked as a change because one of 9/11/2023 3:39:38 PM, 9/18/2023 8:11:22 PM falls between 9/9/2023 9:29:58 PM and 9/18/2023 8:11:22 PM.
IsRegressionStdDev: Marked as regression because -313.31709741786835 (T) = (0 -152.43796767148626) / Math.Sqrt((0.7156885955736987 / (10)) + (0.3166150960668001 / (25))) is less than -2.034515297446192 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (10) + (25) - 2, .025) and -1.4785073222705407 = (61.50394082025106 - 152.43796767148626) / 61.50394082025106 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Memory.ReadOnlySpan.IndexOfString(input: "foobardzsdzs", value: "rddzs", comparisonType: InvariantCulture)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 633.1044664250325 > 134.66019282975518.
IsChangePoint: Marked as a change because one of 9/11/2023 3:39:38 PM, 9/18/2023 8:11:22 PM falls between 9/9/2023 9:29:58 PM and 9/18/2023 8:11:22 PM.
IsRegressionStdDev: Marked as regression because -1200.618598431751 (T) = (0 -634.1729067830589) / Math.Sqrt((1.2179971058865602 / (10)) + (1.277269214203079 / (23))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (10) + (23) - 2, .025) and -3.932085346882248 = (128.58108937306667 - 634.1729067830589) / 128.58108937306667 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Memory.ReadOnlySpan.IndexOfString(input: "だ", value: "た", comparisonType: InvariantCulture)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 781.5508629735062 > 688.1121781352754.
IsChangePoint: Marked as a change because one of 9/11/2023 3:39:38 PM, 9/18/2023 8:11:22 PM falls between 9/9/2023 9:29:58 PM and 9/18/2023 8:11:22 PM.
IsRegressionStdDev: Marked as regression because -21.14154136694366 (T) = (0 -797.3621199610847) / Math.Sqrt((305.53971360814336 / (10)) + (321.848266325625 / (23))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (10) + (23) - 2, .025) and -0.215017997911237 = (656.2553981355393 - 797.3621199610847) / 656.2553981355393 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Memory.ReadOnlySpan.IndexOfString(input: "AAAAA5AAAA", value: "5", comparisonType: InvariantCulture)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 131.98635024959853 > 87.8539989302528.
IsChangePoint: Marked as a change because one of 9/11/2023 8:56:48 PM, 9/18/2023 8:11:22 PM falls between 9/9/2023 9:29:58 PM and 9/18/2023 8:11:22 PM.
IsRegressionStdDev: Marked as regression because -176.61774654652282 (T) = (0 -131.8386915510127) / Math.Sqrt((0.7781561531188188 / (11)) + (0.05745556534874603 / (23))) is less than -2.03693334345674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (11) + (23) - 2, .025) and -0.5687429697853321 = (84.04097681410077 - 131.8386915510127) / 84.04097681410077 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler
Copy link
Author

performanceautofiler bot commented Sep 19, 2023 •
edited
Loading

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline da4e544809b3b10b7db8dc170331988d817240d7
Compare f6c592995a4f8526508da33761230a3850d942ff
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.Globalization.Tests.StringSearch

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
2.51 μs 2.91 μs 1.16 0.01 True
2.13 μs 2.53 μs 1.19 0.01 True
6.36 μs 7.08 μs 1.11 0.03 True
2.12 μs 2.55 μs 1.20 0.01 True
2.51 μs 2.92 μs 1.17 0.01 True
2.51 μs 2.92 μs 1.16 0.01 True
2.12 μs 2.55 μs 1.20 0.01 True
6.36 μs 7.37 μs 1.16 0.04 True
6.47 μs 7.15 μs 1.11 0.03 True
2.13 μs 2.53 μs 1.19 0.01 True
2.14 μs 2.54 μs 1.19 0.01 True
2.12 μs 2.54 μs 1.20 0.01 True
2.51 μs 2.92 μs 1.17 0.01 True
1.18 μs 1.34 μs 1.14 0.02 True
6.50 μs 7.09 μs 1.09 0.02 True

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Globalization.Tests.StringSearch*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Globalization.Tests.StringSearch* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Globalization.Tests.StringSearch*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Globalization.Tests.StringSearch* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

System.Globalization.Tests.StringSearch.LastIndexOf_Word_NotFound(Options: (en-US, IgnoreCase, False))

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.9092567637306193 > 2.612581583357855.
IsChangePoint: Marked as a change because one of 9/11/2023 3:39:38 PM, 9/18/2023 8:11:22 PM falls between 9/9/2023 9:29:58 PM and 9/18/2023 8:11:22 PM.
IsRegressionStdDev: Marked as regression because -69.46410260128376 (T) = (0 -2920.249033536977) / Math.Sqrt((357.285782202992 / (10)) + (52.423763306401725 / (24))) is less than -2.03693334345674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (10) + (24) - 2, .025) and -0.17159820963214492 = (2492.5345647752983 - 2920.249033536977) / 2492.5345647752983 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Globalization.Tests.StringSearch.LastIndexOf_Word_NotFound(Options: (, None, False))

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.5296756005725665 > 2.2101371177302207.
IsChangePoint: Marked as a change because one of 9/11/2023 3:39:38 PM, 9/18/2023 8:11:22 PM falls between 9/9/2023 9:29:58 PM and 9/18/2023 8:11:22 PM.
IsRegressionStdDev: Marked as regression because -42.81114318971673 (T) = (0 -2541.157725967644) / Math.Sqrt((984.481716578311 / (10)) + (74.59756813563875 / (23))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (10) + (23) - 2, .025) and -0.2046595143153932 = (2109.4406309585174 - 2541.157725967644) / 2109.4406309585174 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Globalization.Tests.StringSearch.IndexOf_Word_NotFound(Options: (, IgnoreCase, True))

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 7.082922221592626 > 6.700454377215317.
IsChangePoint: Marked as a change because one of 9/11/2023 3:39:38 PM, 9/18/2023 8:11:22 PM falls between 9/9/2023 9:29:58 PM and 9/18/2023 8:11:22 PM.
IsRegressionStdDev: Marked as regression because -10.690694533221809 (T) = (0 -6865.321776944237) / Math.Sqrt((12299.819062141412 / (10)) + (10048.187734657236 / (24))) is less than -2.03693334345674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (10) + (24) - 2, .025) and -0.06749572114998621 = (6431.240557618722 - 6865.321776944237) / 6431.240557618722 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Globalization.Tests.StringSearch.IndexOf_Word_NotFound(Options: (en-US, IgnoreNonSpace, False))

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.5540082223082345 > 2.203537205577737.
IsChangePoint: Marked as a change because one of 9/11/2023 3:39:38 PM, 9/18/2023 8:11:22 PM falls between 9/9/2023 9:29:58 PM and 9/18/2023 8:11:22 PM.
IsRegressionStdDev: Marked as regression because -48.17405623721936 (T) = (0 -2544.491554563871) / Math.Sqrt((814.8601688380878 / (10)) + (63.94107799043977 / (25))) is less than -2.034515297446192 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (10) + (25) - 2, .025) and -0.21001799566489374 = (2102.854307687958 - 2544.491554563871) / 2102.854307687958 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Globalization.Tests.StringSearch.IndexOf_Word_NotFound(Options: (en-US, IgnoreCase, False))

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.9210979057264317 > 2.6110829737554235.
IsChangePoint: Marked as a change because one of 9/11/2023 3:39:38 PM, 9/18/2023 8:11:22 PM falls between 9/9/2023 9:29:58 PM and 9/18/2023 8:11:22 PM.
IsRegressionStdDev: Marked as regression because -68.89126952178131 (T) = (0 -2923.009824902341) / Math.Sqrt((366.4246856675681 / (10)) + (79.59672576066033 / (24))) is less than -2.03693334345674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (10) + (24) - 2, .025) and -0.17506678791980304 = (2487.5265431311236 - 2923.009824902341) / 2487.5265431311236 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Globalization.Tests.StringSearch.LastIndexOf_Word_NotFound(Options: (, IgnoreCase, False))

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.915635453771404 > 2.615761488394124.
IsChangePoint: Marked as a change because one of 9/11/2023 3:39:38 PM, 9/18/2023 8:11:22 PM falls between 9/9/2023 9:29:58 PM and 9/18/2023 8:11:22 PM.
IsRegressionStdDev: Marked as regression because -72.1923468924096 (T) = (0 -2920.0085083787794) / Math.Sqrt((336.92514286989774 / (10)) + (32.46621129957934 / (23))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (10) + (23) - 2, .025) and -0.17162221039343972 = (2492.2782126144725 - 2920.0085083787794) / 2492.2782126144725 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Globalization.Tests.StringSearch.IndexOf_Word_NotFound(Options: (, None, False))

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.5463668793834615 > 2.2027947232803986.
IsChangePoint: Marked as a change because one of 9/11/2023 3:39:38 PM, 9/18/2023 8:11:22 PM falls between 9/9/2023 9:29:58 PM and 9/18/2023 8:11:22 PM.
IsRegressionStdDev: Marked as regression because -46.694524542416175 (T) = (0 -2544.2368166862275) / Math.Sqrt((866.4158198775918 / (10)) + (71.41527229276647 / (24))) is less than -2.03693334345674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (10) + (24) - 2, .025) and -0.21027533712985344 = (2102.196697422456 - 2544.2368166862275) / 2102.196697422456 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Globalization.Tests.StringSearch.IndexOf_Word_NotFound(Options: (en-US, IgnoreCase, True))

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 7.373017021602944 > 6.684132444850537.
IsChangePoint: Marked as a change because one of 9/11/2023 8:56:48 PM, 9/18/2023 8:11:22 PM falls between 9/9/2023 9:29:58 PM and 9/18/2023 8:11:22 PM.
IsRegressionStdDev: Marked as regression because -13.669649885989287 (T) = (0 -6945.95658074017) / Math.Sqrt((5768.100039665978 / (11)) + (22809.645348065253 / (23))) is less than -2.03693334345674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (11) + (23) - 2, .025) and -0.08298739728684776 = (6413.700286948412 - 6945.95658074017) / 6413.700286948412 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Globalization.Tests.StringSearch.IndexOf_Word_NotFound(Options: (, None, True))

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 7.1509463162495095 > 6.7814070770151895.
IsChangePoint: Marked as a change because one of 9/11/2023 8:56:48 PM, 9/18/2023 8:11:22 PM falls between 9/9/2023 9:29:58 PM and 9/18/2023 8:11:22 PM.
IsRegressionStdDev: Marked as regression because -11.05719483423222 (T) = (0 -7044.8042905126995) / Math.Sqrt((17567.997984381414 / (11)) + (15108.197176073845 / (24))) is less than -2.034515297446192 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (11) + (24) - 2, .025) and -0.0799862338356395 = (6523.050081381714 - 7044.8042905126995) / 6523.050081381714 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Globalization.Tests.StringSearch.LastIndexOf_Word_NotFound(Options: (en-US, None, False))

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.5324267791917907 > 2.204590125327938.
IsChangePoint: Marked as a change because one of 9/11/2023 8:56:48 PM, 9/18/2023 8:11:22 PM falls between 9/9/2023 9:29:58 PM and 9/18/2023 8:11:22 PM.
IsRegressionStdDev: Marked as regression because -44.426303828347685 (T) = (0 -2543.5346703220616) / Math.Sqrt((1018.7965787899835 / (11)) + (58.14302552843269 / (24))) is less than -2.034515297446192 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (11) + (24) - 2, .025) and -0.2052220026046145 = (2110.428340027986 - 2543.5346703220616) / 2110.428340027986 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Globalization.Tests.StringSearch.LastIndexOf_Word_NotFound(Options: (en-US, IgnoreNonSpace, False))

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.535153114369027 > 2.2138581010429883.
IsChangePoint: Marked as a change because one of 9/11/2023 8:56:48 PM, 9/18/2023 8:11:22 PM falls between 9/9/2023 9:29:58 PM and 9/18/2023 8:11:22 PM.
IsRegressionStdDev: Marked as regression because -46.45492895725801 (T) = (0 -2542.5969550117384) / Math.Sqrt((912.2698796621759 / (11)) + (41.68691092566082 / (24))) is less than -2.034515297446192 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (11) + (24) - 2, .025) and -0.2020971072686786 = (2115.134409389646 - 2542.5969550117384) / 2115.134409389646 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Globalization.Tests.StringSearch.IndexOf_Word_NotFound(Options: (en-US, None, False))

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.5443842915565056 > 2.201903185460468.
IsChangePoint: Marked as a change because one of 9/11/2023 8:56:48 PM, 9/18/2023 8:11:22 PM falls between 9/9/2023 9:29:58 PM and 9/18/2023 8:11:22 PM.
IsRegressionStdDev: Marked as regression because -56.40941237881503 (T) = (0 -2543.8937835928214) / Math.Sqrt((659.1885912087879 / (11)) + (36.82811407731527 / (24))) is less than -2.034515297446192 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (11) + (24) - 2, .025) and -0.21042033425619158 = (2101.6614737855134 - 2543.8937835928214) / 2101.6614737855134 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Globalization.Tests.StringSearch.IndexOf_Word_NotFound(Options: (, IgnoreCase, False))

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.9226622969468252 > 2.6087724523826146.
IsChangePoint: Marked as a change because one of 9/11/2023 8:56:48 PM, 9/18/2023 8:11:22 PM falls between 9/9/2023 9:29:58 PM and 9/18/2023 8:11:22 PM.
IsRegressionStdDev: Marked as regression because -73.10054057002459 (T) = (0 -2921.6024856659424) / Math.Sqrt((375.56103421635885 / (11)) + (21.640219579954998 / (24))) is less than -2.034515297446192 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (11) + (24) - 2, .025) and -0.17386957505186573 = (2488.86464711112 - 2921.6024856659424) / 2488.86464711112 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Globalization.Tests.StringSearch.LastIndexOf_Word_NotFound(Options: (en-US, OrdinalIgnoreCase, False))

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.3417236744070058 > 1.208149765164575.
IsChangePoint: Marked as a change because one of 9/13/2023 5:14:37 AM, 9/18/2023 8:11:22 PM falls between 9/9/2023 9:29:58 PM and 9/18/2023 8:11:22 PM.
IsRegressionStdDev: Marked as regression because -27.670462948964808 (T) = (0 -1334.258325285656) / Math.Sqrt((573.2231865011601 / (16)) + (36.782714930950995 / (19))) is less than -2.034515297446192 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (19) - 2, .025) and -0.14605315884395806 = (1164.2202763365212 - 1334.258325285656) / 1164.2202763365212 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Globalization.Tests.StringSearch.IndexOf_Word_NotFound(Options: (en-US, None, True))

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 7.0940247729518155 > 6.758120371595836.
IsChangePoint: Marked as a change because one of 9/11/2023 8:56:48 PM, 9/18/2023 8:11:22 PM falls between 9/9/2023 9:29:58 PM and 9/18/2023 8:11:22 PM.
IsRegressionStdDev: Marked as regression because -11.008817864876569 (T) = (0 -7019.335931989459) / Math.Sqrt((18550.1965263901 / (11)) + (11489.521892970426 / (23))) is less than -2.03693334345674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (11) + (23) - 2, .025) and -0.07912901312360164 = (6504.630907542355 - 7019.335931989459) / 6504.630907542355 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Sep 19, 2023 •
edited
Loading

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 9d08b24d743d0c57203a55c3d0c6dd1bc472e57e
Compare f6c592995a4f8526508da33761230a3850d942ff
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.Collections.Tests.Perf_BitArray

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
77.58 ns 111.93 ns 1.44 0.15 False
8.39 μs 12.52 μs 1.49 0.23 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Collections.Tests.Perf_BitArray*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Collections.Tests.Perf_BitArray* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Collections.Tests.Perf_BitArray*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Collections.Tests.Perf_BitArray* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

System.Collections.Tests.Perf_BitArray.BitArrayGet(Size: 4)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 111.9325973747478 > 91.9642397071438.
IsChangePoint: Marked as a change because one of 9/11/2023 8:56:48 PM, 9/18/2023 8:11:22 PM falls between 9/9/2023 9:29:58 PM and 9/18/2023 8:11:22 PM.
IsRegressionStdDev: Marked as regression because -7.366044017689988 (T) = (0 -108.14086905699908) / Math.Sqrt((102.9482874621626 / (11)) + (13.133497776349907 / (24))) is less than -2.034515297446192 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (11) + (24) - 2, .025) and -0.2728903767160478 = (84.95693818975488 - 108.14086905699908) / 84.95693818975488 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Collections.Tests.Perf_BitArray.BitArrayGet(Size: 512)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 12.518542315384614 > 10.055623598446726.
IsChangePoint: Marked as a change because one of 9/11/2023 3:39:38 PM, 9/18/2023 8:11:22 PM falls between 9/9/2023 9:29:58 PM and 9/18/2023 8:11:22 PM.
IsRegressionStdDev: Marked as regression because -5.994159953429623 (T) = (0 -12099.830329494067) / Math.Sqrt((1913975.032897243 / (10)) + (204438.76371708084 / (23))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (10) + (23) - 2, .025) and -0.284859022751298 = (9417.243538193337 - 12099.830329494067) / 9417.243538193337 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline da4e544809b3b10b7db8dc170331988d817240d7
Compare f6c592995a4f8526508da33761230a3850d942ff
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.Collections.Perf_Frozen<ReferenceType>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
7.34 μs 7.71 μs 1.05 0.00 True

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Collections.Perf_Frozen&lt;ReferenceType&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Collections.Perf_Frozen&lt;ReferenceType&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Collections.Perf_Frozen&lt;ReferenceType&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Collections.Perf_Frozen&lt;ReferenceType&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

System.Collections.Perf_Frozen<ReferenceType>.ToFrozenDictionary(Count: 64)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 7.714037988924572 > 7.636789715009754.
IsChangePoint: Marked as a change because one of 9/11/2023 3:39:38 PM, 9/18/2023 8:11:22 PM falls between 9/9/2023 9:29:58 PM and 9/18/2023 8:11:22 PM.
IsRegressionStdDev: Marked as regression because -11.571407181772548 (T) = (0 -7723.067045196483) / Math.Sqrt((14600.201301831206 / (10)) + (5192.207094172414 / (24))) is less than -2.03693334345674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (10) + (24) - 2, .025) and -0.06535432927367826 = (7249.294279830639 - 7723.067045196483) / 7249.294279830639 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@kotlarmilos
Copy link
Member

kotlarmilos commented Sep 20, 2023 •
edited
Loading

Regressions are likely caused by dotnet/runtime#91884. Details are in dotnet/runtime#91884 (comment).

/cc: @MihaZupan

@kotlarmilos kotlarmilos added LinkedFromMaster Is this issue linked from the master perf issue and removed untriaged labels Sep 20, 2023
This was referenced Sep 20, 2023
Closed
Closed
@MihaZupan
Copy link
Member

CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

I take it this is a configuration without hardware intrinsics support? In that case, yes, it's likely dotnet/runtime#91884

@kotlarmilos
Copy link
Member

Thanks, I think SIMD intrinsics should be enabled on Mono AOT.

/cc: @fanyang-mono @matouskozak

@matouskozak
Copy link
Member

I think that by default SIMD should be enabled but I'm unsure of the exact config that is running in the lab.

@fanyang-mono
Copy link
Member

fanyang-mono commented Sep 20, 2023 •
edited
Loading

I took a brief look at the PR (dotnet/runtime#91884). The hardware intrinsics required to keep the performance were Sse41 and AdvSimd. For this issue, they are regressions on x64. So only Sse41 is relevant here.

First of all, for Mono, Sse41 is currently only supported by LLVM codegen engine.

Secondly, the microbenchmark tests were set up to run on LLVM AOT with JIT fall back. If the methods were AOT'ed with LLVM, then yes, the hardware intrinsics would have been emitted.

However, Mono AOT compiler in normal mode doesn't compile everything, generics is one of them. System.Collections.Perf_Frozen<ReferenceType> clearly uses generics. Thus, it fell back to JIT, where LLVM was not supported. As a result, the expected Sse41 intrinsics weren't emitted.

I believe the rest of the microbenchmarks fell into the same category, where they have code patterns that Mono AOT compiler doesn't compile in normal mode.

I anticipate similar issue will be reported on arm64, where AdvSimd is only supported for Full AOT with LLVM (not the mode that the microbenchmarks run with).

Because of the current limitation of Mono, I suggest that dotnet/runtime#91884 should be reverted or add it back for Mono only.

@MihaZupan
Copy link
Member

MihaZupan commented Sep 20, 2023 •
edited
Loading

The before/after dotnet/runtime#91884 alters which scalar loop we're using:

With the loops being almost the same, the only difference is in the Contains method we call for each position:

In my testing with RyuJIT on x64, the throughput of the two was almost identical.

However, looking at a benchmark like System.Memory.ReadOnlySpan.IndexOfString(input: "string1", value: "string2", comparisonType: InvariantCulture) in this issue, the difference reported here is 7x.

Is Mono running into some horrible codegen issue for the after case?
I could expect some difference, but something is likely going wrong to get to 7x.

@SamMonoRT
Copy link
Member

/cc @jeffhandley

@MihaZupan @kotlarmilos @fanyang-mono - It is a little confusing from comments in the PR itself. Can you confirm this and other Mono config regressions are caused by dotnet/runtime#91884 OR dotnet/runtime#91887 ?

We need to decide to revert OR special case for the regressed Mono codegen path immediately. If any of the PRs were backported, we need to make sure that is also addressed.

@MihaZupan
Copy link
Member

The regressions are about dotnet/runtime#91884. dotnet/runtime#91887 is unrelated. Nothing was backported, we're talking about 9.0 main only.

It would be good to understand why this is happening - #21818 (comment).

@fanyang-mono
Copy link
Member

I will gather the generated code and report back later.

@kotlarmilos
Copy link
Member

Impact of this regression was across the Mono engines:

@kotlarmilos kotlarmilos changed the title [Perf] Linux/x64: 24 Regressions on 9/12/2023 12:17:30 AM [Perf] Mono regression in IndexOf-based search using Contains method Sep 21, 2023
@fanyang-mono
Copy link
Member

fanyang-mono commented Sep 25, 2023 •
edited
Loading

@kotlarmilos help me confirmed that it was dotnet/runtime#91884, which caused this regression.

According to @MihaZupan's analysis of the library code change. It boiled down to the difference between System.Buffers.BitVector256:Contains256 (char) and System.Buffers.ProbabilisticMap:Contains (System.ReadOnlySpan`1,char)

The IL code for them are

method to IR System.Buffers.BitVector256:Contains256 (char)
converting (in B3: stack: 0) IL_0000: ldarg.1   
converting (in B3: stack: 1) IL_0001: ldc.i4    256
converting (in B3: stack: 2) IL_0006: bge.s     IL_0011
converting (in B5: stack: 0) IL_0008: ldarg.0   
converting (in B5: stack: 1) IL_0009: ldarg.1   
converting (in B5: stack: 2) IL_000a: call      0x0600370f
cmethod = bool System.Buffers.BitVector256:ContainsUnchecked (int)
converting (in B5: stack: 1) IL_000f: br.s      IL_0012

method to IR System.Buffers.ProbabilisticMap:Contains (System.ReadOnlySpan`1<char>,char)
converting (in B3: stack: 0) IL_0000: ldarg.0   
converting (in B3: stack: 1) IL_0001: call      0x2b00015e
cmethod = char& System.Runtime.InteropServices.MemoryMarshal:GetReference<char> (System.ReadOnlySpan`1<char>)
converting (in B3: stack: 1) IL_0006: call      0x2b000183
cmethod = int16& System.Runtime.CompilerServices.Unsafe:As<char, int16> (char&)
converting (in B3: stack: 1) IL_000b: ldarg.1   
converting (in B3: stack: 2) IL_000c: conv.i2   
converting (in B3: stack: 2) IL_000d: ldarga.s  0
converting (in B3: stack: 3) IL_000f: call      0x0a000007
cmethod = int System.ReadOnlySpan`1<char>:get_Length ()
converting (in B3: stack: 3) IL_0014: call      0x2b0007d1
cmethod = bool System.SpanHelpers:NonPackedContainsValueType<int16> (int16&,int16,int)
converting (in B3: stack: 1) IL_0019: ret

Then I dived into the calls these two methods made and found out that System.SpanHelpers:NonPackedContainsValueType<int16> (int16&,int16,int) (https://github.com/dotnet/runtime/blob/main/src/libraries/System.Private.CoreLib/src/System/SpanHelpers.T.cs#L1318) is a very expensive call, which eventually caused the regression.

The next piece of the puzzle is that why the generated code for that method from Mono is so slow. I noticed that that method contains vectorization code for Vector128, which Mono has intrinsics support across all codegen engine.

I will report back when I have more information about that.

@MihaZupan
Copy link
Member

MihaZupan commented Sep 25, 2023 •
edited
Loading

Correction to my description in #21818 (comment):

I was describing how the change in dotnet/runtime#91884 would impact IndexOfAny paths (given that the regressed benchmark is IndexOfString). But the code that actually ends up being used here is IndexOfAnyExcept, which uses a meaningfully different code path for the probabilistic case.

Before the change, it would be using BitVector256.Contains256(char) as described above, but after the change, it would be using ProbabilisticMap.IndexOfAnySimpleLoop. This path is different as it effectively does an O(n * m) search, which is bound to be noticeably slower if there are many values present, as is the case here.
The performance here hasn't really been a concern before, as we would normally pick the really fast ASCII implementation that currently works when X86/Arm64/Wasm intrinsics are available (but not "plain Vector128").

With that, even if Mono's execution of NonPackedContainsValueType was as optimal as possible, it would still see a noticeable regression here.

Given the above, I think we should make sure IndexOfAnyAsciiSearcher is supported everywhere we care about perf. This should be the real long-term solution.

We could revert dotnet/runtime#91884 until we can do so, as it may not be trivial.

@fanyang-mono
Copy link
Member

To make it clear, Mono would need to make IndexOfAnyAsciiSearcher.IsVectorizationSupported return true across all the codegen engine to not experience this regression without reverting dotnet/runtime#91884

And the definition of IndexOfAnyAsciiSearcher.IsVectorizationSupported is
https://github.com/dotnet/runtime/blob/aec06846449c331532022b9c01874c80e5a35fc6/src/libraries/System.Private.CoreLib/src/System/SearchValues/IndexOfAnyAsciiSearcher.cs#L35

As I mentioned in my original comment, the current support for Ssse3 and AdvSimd.Arm64 are limited.

@MihaZupan
Copy link
Member

To clarify, supporting IndexOfAnyAsciiSearcher doesn't necessarily mean that Ssse3/AdmSimd.Arm64 should always be supported.
IsVectorizationSupported just reflects what the current implementation supports. If mono always supports Vector128, we could make sure that the implementation has paths for Vector128 in core places like here and here, then flip IsVectorizationSupported to be just Vector128.IsHardwareAccelerated.

@fanyang-mono
Copy link
Member

fanyang-mono commented Sep 26, 2023 •
edited
Loading

Mono indeed supports Vector128 cross all codegen engine. Could you implement that to see if it will eliminate the regression here?

@MihaZupan MihaZupan mentioned this issue Sep 27, 2023
Closed
@fanyang-mono
Copy link
Member

@miha and I discussed offline. We've reached an agreement that his original PR (dotnet/runtime#91884) will be reverted now. And he will be working on adding the Vector128 codepath and make sure it doesn't regress Mono.

@MihaZupan MihaZupan mentioned this issue Sep 27, 2023
Merged
@MihaZupan
Copy link
Member

I'll run benchmarks on dotnet/runtime#92680, but the performance will depend on Mono's implementations of Vector128 support. I'll ping you when I have some numbers.

@matouskozak
Copy link
Member

After the revert dotnet/runtime#92726 we can see the regressions disappeared for AOT #22587.

@matouskozak matouskozak mentioned this issue Oct 16, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kotlarmilos kotlarmilos

@MihaZupan MihaZupan

@fanyang-mono fanyang-mono

Labels
arch-x64 branch-refs/heads/main kind-micro_mono LinkedFromMaster Is this issue linked from the master perf issue mono-aot os-linux perf-regression PGO Applied if there were any profile guided optimization updates in the observed interval. runtime-mono
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@kotlarmilos @MihaZupan @SamMonoRT @fanyang-mono @matouskozak