Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Linux/arm64: 23 Regressions on 3/2/2023 11:02:42 PM #13770

Closed
performanceautofiler bot opened this issue Mar 8, 2023 · 8 comments
Closed

[Perf] Linux/arm64: 23 Regressions on 3/2/2023 11:02:42 PM #13770

performanceautofiler bot opened this issue Mar 8, 2023 · 8 comments
Assignees
@jkotas @kotlarmilos
Labels
ampere arch-arm64 branch-refs/heads/main kind-micro_mono LinkedFromMaster Is this issue linked from the master perf issue mono-aot os-linux perf-regression runtime-mono

Comments

@performanceautofiler
Copy link

performanceautofiler bot commented Mar 8, 2023 •
edited
Loading

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline f2ac74bf23663b7ced738f0f363f97304a2e756e
Compare d2f87f4143b41dfb0ca8f99a50d639702b9aaaf5
Diff Diff

Regressions in System.Linq.Tests.Perf_Enumerable

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
WhereAny_LastElementMatches - Duration of single invocation 1.32 μs 1.41 μs 1.07 0.02 False
Average - Duration of single invocation 1.00 μs 1.08 μs 1.08 0.01 False
Min - Duration of single invocation 1.02 μs 1.09 μs 1.07 0.00 False
FirstWithPredicate_LastElementMatches - Duration of single invocation 1.20 μs 1.30 μs 1.09 0.03 False
SingleWithPredicate_FirstElementMatches - Duration of single invocation 1.20 μs 1.28 μs 1.07 0.03 False
Count - Duration of single invocation 515.41 ns 570.66 ns 1.11 0.18 False
SingleWithPredicate_LastElementMatches - Duration of single invocation 1.22 μs 1.31 μs 1.07 0.02 False
Max - Duration of single invocation 1.02 μs 1.11 μs 1.08 0.01 False
Aggregate - Duration of single invocation 1.18 μs 1.26 μs 1.07 0.03 False
Concat_TenTimes - Duration of single invocation 23.64 μs 25.03 μs 1.06 0.00 False
SequenceEqual - Duration of single invocation 2.23 μs 2.38 μs 1.07 0.00 False
WhereFirst_LastElementMatches - Duration of single invocation 1.31 μs 1.40 μs 1.07 0.01 False
AnyWithPredicate_LastElementMatches - Duration of single invocation 1.21 μs 1.33 μs 1.10 0.05 False
Skip_One - Duration of single invocation 2.31 μs 2.46 μs 1.06 0.00 False
All_AllElementsMatch - Duration of single invocation 1.22 μs 1.29 μs 1.06 0.04 False
Aggregate_Seed - Duration of single invocation 1.20 μs 1.28 μs 1.07 0.01 False
Sum - Duration of single invocation 1.14 μs 1.20 μs 1.06 0.00 False
WhereSingle_LastElementMatches - Duration of single invocation 1.33 μs 1.41 μs 1.06 0.01 False
SingleWithPredicate_LastElementMatches - Duration of single invocation 1.22 μs 1.30 μs 1.07 0.04 False
WhereSingleOrDefault_LastElementMatches - Duration of single invocation 1.34 μs 1.41 μs 1.06 0.02 False
WhereLast_LastElementMatches - Duration of single invocation 1.33 μs 1.41 μs 1.06 0.01 False
Contains_ElementNotFound - Duration of single invocation 1.32 μs 1.39 μs 1.06 0.00 False
SingleWithPredicate_FirstElementMatches - Duration of single invocation 1.20 μs 1.27 μs 1.06 0.02 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Linq.Tests.Perf_Enumerable*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Linq.Tests.Perf_Enumerable* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Linq.Tests.Perf_Enumerable*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Linq.Tests.Perf_Enumerable* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Linq.Tests.Perf_Enumerable.WhereAny_LastElementMatches(input: IEnumerable)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.4100617137151674 > 1.3912754264302953.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/2/2023 6:22:31 PM, 3/7/2023 9:28:02 PM falls between 2/27/2023 8:52:52 AM and 3/7/2023 9:28:02 PM.
IsRegressionStdDev: Marked as regression because -20.335570028283133 (T) = (0 -1407.5933025737643) / Math.Sqrt((179.06798526915375 / (23)) + (127.30537244516937 / (19))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (19) - 2, .025) and -0.0581852372186088 = (1330.1955584577597 - 1407.5933025737643) / 1330.1955584577597 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.Average(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.0805611844237626 > 1049.685872954852.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/2/2023 6:22:31 PM, 3/7/2023 9:28:02 PM falls between 2/27/2023 8:52:52 AM and 3/7/2023 9:28:02 PM.
IsRegressionStdDev: Marked as regression because -21.502818160941718 (T) = (0 -1081.9773048239765) / Math.Sqrt((280.4794402390143 / (23)) + (2.961352579552723 / (19))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (19) - 2, .025) and -0.07508708355225918 = (1006.4089889806423 - 1081.9773048239765) / 1006.4089889806423 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.Min(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.0894356950304733 > 1.0608053431529545.
IsChangePoint: Marked as a change because one of 3/2/2023 6:22:31 PM, 3/7/2023 9:28:02 PM falls between 2/27/2023 8:52:52 AM and 3/7/2023 9:28:02 PM.
IsRegressionStdDev: Marked as regression because -25.842188953246694 (T) = (0 -1090.0566116610821) / Math.Sqrt((172.5451934219008 / (23)) + (1.7113395873656077 / (19))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (19) - 2, .025) and -0.06988714975277947 = (1018.8519526689925 - 1090.0566116610821) / 1018.8519526689925 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.FirstWithPredicate_LastElementMatches(input: Array)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.2997866750208853 > 1.2677539197955952.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/1/2023 10:50:04 PM, 3/7/2023 9:28:02 PM falls between 2/27/2023 8:52:52 AM and 3/7/2023 9:28:02 PM.
IsRegressionStdDev: Marked as regression because -24.472756883398002 (T) = (0 -1286.0778637835201) / Math.Sqrt((48.898682866744835 / (20)) + (157.94554132866915 / (22))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (22) - 2, .025) and -0.0627372674158019 = (1210.1559841886442 - 1286.0778637835201) / 1210.1559841886442 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.SingleWithPredicate_FirstElementMatches(input: Array)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.2799567276290007 > 1.2558502503344633.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/2/2023 6:22:31 PM, 3/7/2023 9:28:02 PM falls between 2/27/2023 8:52:52 AM and 3/7/2023 9:28:02 PM.
IsRegressionStdDev: Marked as regression because -17.49715567891471 (T) = (0 -1281.194139175015) / Math.Sqrt((172.65219020358703 / (23)) + (245.3816149010689 / (19))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (19) - 2, .025) and -0.06577505701307663 = (1202.1243420404942 - 1281.194139175015) / 1202.1243420404942 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.Count(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 570.6589722585271 > 544.7699132780987.
IsChangePoint: Marked as a change because one of 3/1/2023 10:50:04 PM, 3/7/2023 9:28:02 PM falls between 2/27/2023 8:52:52 AM and 3/7/2023 9:28:02 PM.
IsRegressionStdDev: Marked as regression because -12.718456637450243 (T) = (0 -572.0504114535192) / Math.Sqrt((122.53697683523527 / (20)) + (215.02836114323867 / (22))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (22) - 2, .025) and -0.09728107938392042 = (521.33443490587 - 572.0504114535192) / 521.33443490587 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.SingleWithPredicate_LastElementMatches(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.3072749137521356 > 1.2769915719245697.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/2/2023 4:26:55 AM, 3/7/2023 9:28:02 PM falls between 2/27/2023 8:52:52 AM and 3/7/2023 9:28:02 PM.
IsRegressionStdDev: Marked as regression because -29.236393924654703 (T) = (0 -1297.5099086201506) / Math.Sqrt((81.98713690270743 / (21)) + (64.01787788163438 / (20))) is less than -2.0226909200346674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (20) - 2, .025) and -0.06389944211500316 = (1219.5794614204667 - 1297.5099086201506) / 1219.5794614204667 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.Max(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.1061397158185236 > 1.0741649798717279.
IsChangePoint: Marked as a change because one of 3/2/2023 4:26:55 AM, 3/7/2023 9:28:02 PM falls between 2/27/2023 8:52:52 AM and 3/7/2023 9:28:02 PM.
IsRegressionStdDev: Marked as regression because -42.16487258321924 (T) = (0 -1107.4836241798473) / Math.Sqrt((76.47190297809051 / (21)) + (2.41958762519313 / (21))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (21) - 2, .025) and -0.07967299665223782 = (1025.758380189041 - 1107.4836241798473) / 1025.758380189041 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.Aggregate(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.2605934894779443 > 1.2339441928265924.
IsChangePoint: Marked as a change because one of 11/11/2022 5:53:56 PM, 1/5/2023 4:05:11 PM, 3/2/2023 4:26:55 AM, 3/7/2023 9:28:02 PM falls between 2/27/2023 8:52:52 AM and 3/7/2023 9:28:02 PM.
IsRegressionStdDev: Marked as regression because -20.60962952741124 (T) = (0 -1270.9282639986136) / Math.Sqrt((89.18754141006266 / (21)) + (329.5180939444137 / (21))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (21) - 2, .025) and -0.07806160602980365 = (1178.9013326233585 - 1270.9282639986136) / 1178.9013326233585 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.Concat_TenTimes(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 25.029088283762057 > 24.80709518126888.
IsChangePoint: Marked as a change because one of 3/2/2023 6:22:31 PM, 3/7/2023 9:28:02 PM falls between 2/27/2023 8:52:52 AM and 3/7/2023 9:28:02 PM.
IsRegressionStdDev: Marked as regression because -25.14424427120247 (T) = (0 -25037.053511793685) / Math.Sqrt((42973.63947110707 / (23)) + (15833.474660107937 / (19))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (19) - 2, .025) and -0.05507601291787405 = (23730.094519495575 - 25037.053511793685) / 23730.094519495575 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.SequenceEqual(input1: IEnumerable, input2: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.3822457350447115 > 2.3463670317398306.
IsChangePoint: Marked as a change because one of 3/1/2023 10:50:04 PM, 3/7/2023 9:28:02 PM falls between 2/27/2023 8:52:52 AM and 3/7/2023 9:28:02 PM.
IsRegressionStdDev: Marked as regression because -28.414681040468253 (T) = (0 -2391.642570885394) / Math.Sqrt((178.13051775491874 / (20)) + (423.7753139155936 / (22))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (22) - 2, .025) and -0.06730065824180956 = (2240.8330327699837 - 2391.642570885394) / 2240.8330327699837 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.WhereFirst_LastElementMatches(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.4003928187646741 > 1.3776051683457218.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/2/2023 6:22:31 PM, 3/7/2023 9:28:02 PM falls between 2/27/2023 8:52:52 AM and 3/7/2023 9:28:02 PM.
IsRegressionStdDev: Marked as regression because -24.25571996880671 (T) = (0 -1399.3168187241815) / Math.Sqrt((214.09976886018683 / (23)) + (35.732389752738506 / (19))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (19) - 2, .025) and -0.061551911305598866 = (1318.1803017086247 - 1399.3168187241815) / 1318.1803017086247 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.AnyWithPredicate_LastElementMatches(input: Array)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.3262560860504153 > 1.2677685219934975.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/1/2023 10:50:04 PM, 3/7/2023 9:28:02 PM falls between 2/27/2023 8:52:52 AM and 3/7/2023 9:28:02 PM.
IsRegressionStdDev: Marked as regression because -17.201703815488603 (T) = (0 -1292.685891191714) / Math.Sqrt((32.46996973490178 / (20)) + (496.8206048309925 / (22))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (22) - 2, .025) and -0.0700565891108008 = (1208.0537649564071 - 1292.685891191714) / 1208.0537649564071 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.Skip_One(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.4596523002516517 > 2.430588655494419.
IsChangePoint: Marked as a change because one of 3/2/2023 6:22:31 PM, 3/7/2023 9:28:02 PM falls between 2/27/2023 8:52:52 AM and 3/7/2023 9:28:02 PM.
IsRegressionStdDev: Marked as regression because -19.663969385046073 (T) = (0 -2455.481641812855) / Math.Sqrt((814.8541942266912 / (23)) + (112.66742022011212 / (19))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (19) - 2, .025) and -0.05429739708803397 = (2329.0218192655007 - 2455.481641812855) / 2329.0218192655007 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.All_AllElementsMatch(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.286862364453665 > 1.2829352607731042.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/1/2023 10:50:04 PM, 3/7/2023 9:28:02 PM falls between 2/27/2023 8:52:52 AM and 3/7/2023 9:28:02 PM.
IsRegressionStdDev: Marked as regression because -15.023621100659762 (T) = (0 -1300.5774685998042) / Math.Sqrt((55.30325782021996 / (20)) + (611.5783045714226 / (22))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (22) - 2, .025) and -0.06821894909673178 = (1217.519563474839 - 1300.5774685998042) / 1217.519563474839 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.Aggregate_Seed(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.2838242638955077 > 1.2490459217223573.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/1/2023 10:50:04 PM, 3/7/2023 9:28:02 PM falls between 2/27/2023 8:52:52 AM and 3/7/2023 9:28:02 PM.
IsRegressionStdDev: Marked as regression because -28.88025530064354 (T) = (0 -1284.0907817174034) / Math.Sqrt((91.63633991805372 / (20)) + (115.11338138860545 / (22))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (22) - 2, .025) and -0.0757992577502073 = (1193.615604832068 - 1284.0907817174034) / 1193.615604832068 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.Sum(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.203521272184763 > 1.1893851712354324.
IsChangePoint: Marked as a change because one of 3/1/2023 10:50:04 PM, 3/7/2023 9:28:02 PM falls between 2/27/2023 8:52:52 AM and 3/7/2023 9:28:02 PM.
IsRegressionStdDev: Marked as regression because -45.20515300306318 (T) = (0 -1201.5277448582551) / Math.Sqrt((4.145235238352749 / (20)) + (45.213536928476366 / (22))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (22) - 2, .025) and -0.05998471521391383 = (1133.5330855367822 - 1201.5277448582551) / 1133.5330855367822 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.WhereSingle_LastElementMatches(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.4057905645887014 > 1.3922075797185935.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/2/2023 6:22:31 PM, 3/7/2023 9:28:02 PM falls between 2/27/2023 8:52:52 AM and 3/7/2023 9:28:02 PM.
IsRegressionStdDev: Marked as regression because -21.061820431412233 (T) = (0 -1414.0178747096481) / Math.Sqrt((181.55454119426952 / (23)) + (130.94762024568956 / (19))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (19) - 2, .025) and -0.06075416357139116 = (1333.0307089711287 - 1414.0178747096481) / 1333.0307089711287 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.SingleWithPredicate_LastElementMatches(input: Array)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.3022593811202505 > 1.2787063466362671.
IsChangePoint: Marked as a change because one of 11/24/2022 8:21:24 PM, 1/5/2023 11:12:07 AM, 3/1/2023 10:50:04 PM, 3/7/2023 9:28:02 PM falls between 2/27/2023 8:52:52 AM and 3/7/2023 9:28:02 PM.
IsRegressionStdDev: Marked as regression because -26.492907639387635 (T) = (0 -1300.2695917024955) / Math.Sqrt((45.660773585466515 / (20)) + (175.75255414854183 / (22))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (22) - 2, .025) and -0.06986304245864235 = (1215.3607892785585 - 1300.2695917024955) / 1215.3607892785585 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.WhereSingleOrDefault_LastElementMatches(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.4134269025357058 > 1.3931963021100526.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/1/2023 10:50:04 PM, 3/7/2023 9:28:02 PM falls between 2/27/2023 8:52:52 AM and 3/7/2023 9:28:02 PM.
IsRegressionStdDev: Marked as regression because -28.846911702516056 (T) = (0 -1411.6385997273999) / Math.Sqrt((17.21678967551934 / (20)) + (160.79938514214876 / (22))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (22) - 2, .025) and -0.06203291842281312 = (1329.1853531468437 - 1411.6385997273999) / 1329.1853531468437 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.WhereLast_LastElementMatches(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.410254541515289 > 1.4028670999183501.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/2/2023 6:22:31 PM, 3/7/2023 9:28:02 PM falls between 2/27/2023 8:52:52 AM and 3/7/2023 9:28:02 PM.
IsRegressionStdDev: Marked as regression because -15.946272723969567 (T) = (0 -1427.6248955766769) / Math.Sqrt((184.36939186847505 / (23)) + (422.00804389988406 / (19))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (19) - 2, .025) and -0.0654285027155294 = (1339.9537293567735 - 1427.6248955766769) / 1339.9537293567735 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.Contains_ElementNotFound(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.3936444252615843 > 1.389785774750381.
IsChangePoint: Marked as a change because one of 3/1/2023 10:50:04 PM, 3/7/2023 9:28:02 PM falls between 2/27/2023 8:52:52 AM and 3/7/2023 9:28:02 PM.
IsRegressionStdDev: Marked as regression because -61.55328533371159 (T) = (0 -1395.6297782366576) / Math.Sqrt((13.133703696852827 / (20)) + (15.576180752020518 / (22))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (22) - 2, .025) and -0.05432148224111866 = (1323.723173381649 - 1395.6297782366576) / 1323.723173381649 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.SingleWithPredicate_FirstElementMatches(input: IEnumerable)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.2710523686062256 > 1.2585805195720843.
IsChangePoint: Marked as a change because one of 12/21/2022 10:31:08 AM, 3/1/2023 10:50:04 PM, 3/7/2023 9:28:02 PM falls between 2/27/2023 8:52:52 AM and 3/7/2023 9:28:02 PM.
IsRegressionStdDev: Marked as regression because -17.89427008791966 (T) = (0 -1279.0119730575177) / Math.Sqrt((52.68854675638544 / (20)) + (335.78906302848674 / (22))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (22) - 2, .025) and -0.06291202919520386 = (1203.309340685453 - 1279.0119730575177) / 1203.309340685453 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@kotlarmilos
Copy link
Member

The only relevant change that could have created the regressions is dotnet/runtime#82751. Assigning for accuracy tracking.

@kotlarmilos kotlarmilos mentioned this issue Mar 8, 2023
Closed
@kotlarmilos kotlarmilos added the LinkedFromMaster Is this issue linked from the master perf issue label Mar 8, 2023
@stephentoub
Copy link
Member

@kotlarmilos, thanks, is the link wrong or did you mean to assign this to Jan?

@kotlarmilos
Copy link
Member

@kotlarmilos, thanks, is the link wrong or did you mean to assign this to Jan?

Correct, my error, assigning myself as well.

@kotlarmilos kotlarmilos assigned jkotas and kotlarmilos and unassigned stephentoub Mar 8, 2023
@jkotas
Copy link
Member

jkotas commented Mar 8, 2023 •
edited
Loading

The only relevant change that could have created the regressions is dotnet/runtime#82751

This change is not included in the diff. Did you mean to link to dotnet/runtime#82899 that is part of the diff?

@kotlarmilos
Copy link
Member

It seems that range isn't the same in the issue and on the dashboard. For example, there was a regression in System.Linq.Tests.Perf_Enumerable.Count(input: IEnumerable) for dotnet/runtime@8d2ddc4...e303a45 range.

That range doesn't overlap with the range reported in the issue.

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline f2ac74bf23663b7ced738f0f363f97304a2e756e
Compare d2f87f4143b41dfb0ca8f99a50d639702b9aaaf5
Diff Diff

@jkotas
Copy link
Member

jkotas commented Mar 8, 2023

It may be two different regressions. These types of microbenchmarks measure tiny bits of code, and any subtle change in the tiny bit of code tend to manifest as hard to explain set of improvements and/or regressions due to microarchitecture issues.

@kotlarmilos
Copy link
Member

Make sense. I will try to identify the regressions by bisecting the range.

@kotlarmilos
Copy link
Member

According to historical data, it is a transient issue, shouldn't be regression. Closing this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jkotas jkotas

@kotlarmilos kotlarmilos

Labels
ampere arch-arm64 branch-refs/heads/main kind-micro_mono LinkedFromMaster Is this issue linked from the master perf issue mono-aot os-linux perf-regression runtime-mono
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@stephentoub @jkotas @kotlarmilos