Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Linux/x64: 1080 Improvements on 9/26/2023 6:21:18 AM #22688

Closed
performanceautofiler bot opened this issue Oct 3, 2023 · 6 comments
Closed

[Perf] Linux/x64: 1080 Improvements on 9/26/2023 6:21:18 AM #22688

performanceautofiler bot opened this issue Oct 3, 2023 · 6 comments
Labels
arch-x64 branch-refs/heads/main kind-micro_mono mono-jit os-linux perf-improvement PGO Applied if there were any profile guided optimization updates in the observed interval. runtime-mono

Comments

@performanceautofiler
Copy link

performanceautofiler bot commented Oct 3, 2023 •
edited
Loading

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 736dabeca728ccf8b911d96d1b4c575b4d0db7d2
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:false, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Text.Json.Serialization.Tests.WriteJson<BinaryData>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
2.64 μs 2.33 μs 0.88 0.01 True
8.77 μs 7.11 μs 0.81 0.03 False
9.20 μs 7.46 μs 0.81 0.03 False
2.01 μs 1.73 μs 0.86 0.01 True
2.71 μs 2.46 μs 0.91 0.01 True
8.67 μs 7.00 μs 0.81 0.03 False
9.15 μs 7.42 μs 0.81 0.03 False
2.63 μs 2.34 μs 0.89 0.01 True
2.80 μs 2.52 μs 0.90 0.01 True
1.92 μs 1.64 μs 0.86 0.01 True

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;BinaryData&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;BinaryData&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;BinaryData&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;BinaryData&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

System.Text.Json.Serialization.Tests.WriteJson<BinaryData>.SerializeToUtf8Bytes(Mode: SourceGen)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 2.3299402450974758 < 2.4775997958841582.
IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 62.73453858232696 (T) = (0 -2324.189803516809) / Math.Sqrt((427.7704356494315 / (126)) + (515.2610248996824 / (30))) is greater than 1.975488058234171 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (126) + (30) - 2, .975) and 0.10906875778851592 = (2608.719610895749 - 2324.189803516809) / 2608.719610895749 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Text.Json.Serialization.Tests.WriteJson<BinaryData>.SerializeToString(Mode: Reflection)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 7.110975969061745 < 8.348563525675427.
IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 23.163242685168207 (T) = (0 -7097.918055677936) / Math.Sqrt((827229.9513849368 / (124)) + (652.8950609623226 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.21072123906534315 = (8992.916580287356 - 7097.918055677936) / 8992.916580287356 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Text.Json.Serialization.Tests.WriteJson<BinaryData>.SerializeObjectProperty(Mode: SourceGen)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 7.455978238148902 < 8.723190875713346.
IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 22.351029266335757 (T) = (0 -7421.110736123493) / Math.Sqrt((837346.812472396 / (122)) + (16293.345993794796 / (30))) is greater than 1.9759053308964478 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (122) + (30) - 2, .975) and 0.20584624433163493 = (9344.677505022735 - 7421.110736123493) / 9344.677505022735 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Text.Json.Serialization.Tests.WriteJson<BinaryData>.SerializeToWriter(Mode: Reflection)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.7323625391655675 < 1.9111804033902116.
IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 147.79542194186251 (T) = (0 -1731.9700646476024) / Math.Sqrt((138.52816994919186 / (124)) + (65.25687030039691 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.13407735788030786 = (2000.1441011034342 - 1731.9700646476024) / 2000.1441011034342 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Text.Json.Serialization.Tests.WriteJson<BinaryData>.SerializeToUtf8Bytes(Mode: Reflection)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 2.4599284667133303 < 2.575731595163559.
IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 69.01183835993712 (T) = (0 -2431.8552379794096) / Math.Sqrt((352.1105735540241 / (125)) + (380.2610937419662 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.10047466916913002 = (2703.4872222366025 - 2431.8552379794096) / 2703.4872222366025 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Text.Json.Serialization.Tests.WriteJson<BinaryData>.SerializeToString(Mode: SourceGen)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 7.002036002753373 < 8.276224709059404.
IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 23.729203844469122 (T) = (0 -6969.325208595424) / Math.Sqrt((822486.8725176402 / (125)) + (1782.146215243782 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.2171783019585326 = (8902.825798048136 - 6969.325208595424) / 8902.825798048136 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Text.Json.Serialization.Tests.WriteJson<BinaryData>.SerializeObjectProperty(Mode: Reflection)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 7.420879041212264 < 8.696067287317117.
IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 23.317780716536973 (T) = (0 -7460.251721029422) / Math.Sqrt((827673.1569857149 / (125)) + (1800.0477680254462 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.20349567358617096 = (9366.241304197762 - 7460.251721029422) / 9366.241304197762 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Text.Json.Serialization.Tests.WriteJson<BinaryData>.SerializeToStream(Mode: SourceGen)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 2.3374556363233614 < 2.5015579804307237.
IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 81.93519339815794 (T) = (0 -2354.9656834454922) / Math.Sqrt((309.10192310139485 / (124)) + (274.7564679793746 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.10615390143603341 = (2634.6433544084693 - 2354.9656834454922) / 2634.6433544084693 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Text.Json.Serialization.Tests.WriteJson<BinaryData>.SerializeToStream(Mode: Reflection)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 2.5216809139019896 < 2.6650111009001836.
IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 61.92014862712943 (T) = (0 -2538.967820972723) / Math.Sqrt((289.0687760607798 / (125)) + (414.53868484511787 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.08921065635697167 = (2787.6564857656454 - 2538.967820972723) / 2787.6564857656454 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Text.Json.Serialization.Tests.WriteJson<BinaryData>.SerializeToWriter(Mode: SourceGen)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.638764971736627 < 1.8293183416552434.
IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 198.79166515082673 (T) = (0 -1630.7247483009853) / Math.Sqrt((187.36170648535764 / (124)) + (17.43963759475386 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.1498994590298046 = (1918.2728038731582 - 1630.7247483009853) / 1918.2728038731582 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 736dabeca728ccf8b911d96d1b4c575b4d0db7d2
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:false, MonoInterpreter:false, RunKind:micro_mono

Improvements in Microsoft.Extensions.Configuration.ConfigurationBinderBenchmarks

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
16.49 ms 15.52 ms 0.94 0.02 False
3.20 ms 3.04 ms 0.95 0.01 True
73.06 ms 69.00 ms 0.94 0.01 False

graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'Microsoft.Extensions.Configuration.ConfigurationBinderBenchmarks*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'Microsoft.Extensions.Configuration.ConfigurationBinderBenchmarks*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'Microsoft.Extensions.Configuration.ConfigurationBinderBenchmarks*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'Microsoft.Extensions.Configuration.ConfigurationBinderBenchmarks*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Microsoft.Extensions.Configuration.ConfigurationBinderBenchmarks.Get(ConfigurationProvidersCount: 16, KeysCountPerProvider: 40)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 15.515196533653846 < 15.643438930357142.
IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 32.73125113007692 (T) = (0 -15442142.51269536) / Math.Sqrt((52872686184.82406 / (125)) + (8063332844.2215185 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.0528046415801638 = (16303017.508929519 - 15442142.51269536) / 16303017.508929519 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Microsoft.Extensions.Configuration.ConfigurationBinderBenchmarks.Get(ConfigurationProvidersCount: 16, KeysCountPerProvider: 10)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 3.035298787499999 < 3.036913944196429.
IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 40.99122342937593 (T) = (0 -3026928.3808597764) / Math.Sqrt((1434245618.0264914 / (125)) + (169690055.96499655 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.05307462981474758 = (3196585.999451679 - 3026928.3808597764) / 3196585.999451679 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Microsoft.Extensions.Configuration.ConfigurationBinderBenchmarks.Get(ConfigurationProvidersCount: 32, KeysCountPerProvider: 40)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 68.99556689285716 < 69.22119603520834.
IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 33.503028768730196 (T) = (0 -68580064.15452991) / Math.Sqrt((998967850693.7069 / (124)) + (158259201696.67404 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.053394287722418145 = (72448394.57974827 - 68580064.15452991) / 72448394.57974827 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler
Copy link
Author

performanceautofiler bot commented Oct 3, 2023 •
edited
Loading

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 736dabeca728ccf8b911d96d1b4c575b4d0db7d2
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:false, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.IO.Tests.StreamReaderReadToEndTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
336.15 ms 287.25 ms 0.85 0.01 False
337.94 ms 287.89 ms 0.85 0.03 False
331.85 ms 282.51 ms 0.85 0.04 False
333.30 ms 280.59 ms 0.84 0.04 False
332.46 ms 279.87 ms 0.84 0.03 False
331.95 ms 282.06 ms 0.85 0.03 False
338.43 ms 286.78 ms 0.85 0.04 False
331.85 ms 281.88 ms 0.85 0.03 False
340.14 ms 286.26 ms 0.84 0.01 False
332.07 ms 280.73 ms 0.85 0.03 False
337.79 ms 284.96 ms 0.84 0.01 False
334.06 ms 280.43 ms 0.84 0.05 False
338.84 ms 287.81 ms 0.85 0.03 False
339.01 ms 286.79 ms 0.85 0.01 False
336.81 ms 288.23 ms 0.86 0.01 False
331.36 ms 280.42 ms 0.85 0.04 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.IO.Tests.StreamReaderReadToEndTests*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.IO.Tests.StreamReaderReadToEndTests*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.IO.Tests.StreamReaderReadToEndTests*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.IO.Tests.StreamReaderReadToEndTests*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

System.IO.Tests.StreamReaderReadToEndTests.ReadToEndAsync(LineLengthRange: [ 0, 1024])

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 287.25442439999995 < 320.8313500602564.
IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 34.72182300094011 (T) = (0 -286506199.38490844) / Math.Sqrt((331302222687182.6 / (126)) + (1333068901671.0383 / (30))) is greater than 1.975488058234171 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (126) + (30) - 2, .975) and 0.16539279120941652 = (343282680.0047416 - 286506199.38490844) / 343282680.0047416 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.IO.Tests.StreamReaderReadToEndTests.ReadToEndAsync(LineLengthRange: [1025, 2048])

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 287.8888304285714 < 320.26250603904765.
IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 35.42704939987801 (T) = (0 -286665179.90379727) / Math.Sqrt((317750170652286 / (125)) + (1331892848053.7986 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.1657978849315697 = (343639958.1416572 - 286665179.90379727) / 343639958.1416572 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.IO.Tests.StreamReaderReadToEndTests.ReadToEnd(LineLengthRange: [ 33, 128])

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 282.5056788 < 315.1536158148809.
IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 34.20953847590856 (T) = (0 -280849539.2860623) / Math.Sqrt((339200940205468.6 / (124)) + (1346063482288.224 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.16881845635900697 = (337891934.0001224 - 280849539.2860623) / 337891934.0001224 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.IO.Tests.StreamReaderReadToEndTests.ReadToEnd(LineLengthRange: [ 1, 8])

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 280.59326914285714 < 316.1101889983333.
IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 35.34428094335056 (T) = (0 -280908139.1013125) / Math.Sqrt((319742955767988.7 / (125)) + (929781405424.7622 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.16836385847921612 = (337777695.1680162 - 280908139.1013125) / 337777695.1680162 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.IO.Tests.StreamReaderReadToEndTests.ReadToEnd(LineLengthRange: [ 0, 0])

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 279.8717362 < 315.9447551410714.
IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 35.242203292359484 (T) = (0 -280940861.2462271) / Math.Sqrt((314300329223986.2 / (125)) + (2292376149263.037 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.1679936928782689 = (337666744.64058185 - 280940861.2462271) / 337666744.64058185 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.IO.Tests.StreamReaderReadToEndTests.ReadToEnd(LineLengthRange: [1025, 2048])

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 282.0571350714286 < 316.4133689696429.
IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 35.93592694629357 (T) = (0 -280860984.109017) / Math.Sqrt((310383075888907.44 / (125)) + (1425397720285.239 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.1691169670703118 = (338027102.46555763 - 280860984.109017) / 338027102.46555763 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.IO.Tests.StreamReaderReadToEndTests.ReadToEndAsync(LineLengthRange: [ 33, 128])

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 286.77644626666665 < 321.5062350982143.
IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 30.983930555568996 (T) = (0 -287562967.7296886) / Math.Sqrt((322320412385306.9 / (125)) + (21495186810381.75 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.16358946725287724 = (343806009.6938417 - 287562967.7296886) / 343806009.6938417 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.IO.Tests.StreamReaderReadToEndTests.ReadToEnd(LineLengthRange: [ 1, 1])

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 281.8767843076923 < 315.6672173299999.
IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 34.89699433854306 (T) = (0 -280938480.4558913) / Math.Sqrt((325130977324010.94 / (125)) + (1966039347767.3923 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.16863445873656516 = (337924133.86408365 - 280938480.4558913) / 337924133.86408365 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.IO.Tests.StreamReaderReadToEndTests.ReadToEndAsync(LineLengthRange: [ 1, 8])

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 286.2619132857143 < 322.50983253976193.
IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 35.724377260431986 (T) = (0 -286583788.55735654) / Math.Sqrt((311337676902239.25 / (125)) + (1642931766401.7732 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.16588993690156228 = (343580303.4107925 - 286583788.55735654) / 343580303.4107925 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.IO.Tests.StreamReaderReadToEndTests.ReadToEnd(LineLengthRange: [ 9, 32])

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 280.7323643846154 < 316.3590873492308.
IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 34.70694091107197 (T) = (0 -280669471.5585043) / Math.Sqrt((332079396606862.1 / (124)) + (1880056998850.8533 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.169929612080353 = (338127315.0364133 - 280669471.5585043) / 338127315.0364133 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.IO.Tests.StreamReaderReadToEndTests.ReadToEndAsync(LineLengthRange: [ 1, 1])

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 284.9574644 < 321.42003637582417.
IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 35.12758616806547 (T) = (0 -286345831.6484005) / Math.Sqrt((320531196233789.25 / (124)) + (1227675563414.8616 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.16582490093219196 = (343268256.23108673 - 286345831.6484005) / 343268256.23108673 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.IO.Tests.StreamReaderReadToEndTests.ReadToEnd(LineLengthRange: [ 129, 1024])

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 280.4297033333333 < 315.52488862074176.
IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 31.267070944152906 (T) = (0 -282110002.52118427) / Math.Sqrt((322349345504725.3 / (125)) + (18483696966022.88 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.1653496092943174 = (337997808.01955307 - 282110002.52118427) / 337997808.01955307 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.IO.Tests.StreamReaderReadToEndTests.ReadToEndAsync(LineLengthRange: [ 9, 32])

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 287.8063017692308 < 322.0036629957417.
IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 35.10211815845348 (T) = (0 -286576239.7553479) / Math.Sqrt((320364898549288.56 / (124)) + (1704257729804.6577 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.16599543489706872 = (343614713.5717167 - 286576239.7553479) / 343614713.5717167 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.IO.Tests.StreamReaderReadToEndTests.ReadToEndAsync(LineLengthRange: [ 0, 0])

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 286.7882369230769 < 320.96296458999996.
IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 35.02585306109359 (T) = (0 -286757761.7706166) / Math.Sqrt((318605663229696.44 / (124)) + (1702456752188.2666 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.1652335433472207 = (343518548.79321456 - 286757761.7706166) / 343518548.79321456 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.IO.Tests.StreamReaderReadToEndTests.ReadToEndAsync(LineLengthRange: [ 129, 1024])

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 288.2267696923077 < 321.34507123379115.
IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 35.449271121752695 (T) = (0 -286999030.5994689) / Math.Sqrt((306094640230591.8 / (122)) + (1544322026946.815 / (30))) is greater than 1.9759053308964478 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (122) + (30) - 2, .975) and 0.1650277759496884 = (343722847.6981956 - 286999030.5994689) / 343722847.6981956 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.IO.Tests.StreamReaderReadToEndTests.ReadToEnd(LineLengthRange: [ 0, 1024])

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 280.4199202 < 315.16306485538456.
IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 35.583442901068736 (T) = (0 -281030360.7371306) / Math.Sqrt((316982743754415.8 / (126)) + (1931226688560.7769 / (30))) is greater than 1.975488058234171 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (126) + (30) - 2, .975) and 0.16900911267912158 = (338187054.7860938 - 281030360.7371306) / 338187054.7860938 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Oct 3, 2023 •
edited
Loading

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 736dabeca728ccf8b911d96d1b4c575b4d0db7d2
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:false, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Text.Json.Document.Tests.Perf_EnumerateObject

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
23.18 μs 16.40 μs 0.71 0.01 True
2.75 ms 2.38 ms 0.87 0.01 True
22.91 μs 16.13 μs 0.70 0.01 True
10.77 μs 8.33 μs 0.77 0.01 True
22.94 μs 16.14 μs 0.70 0.01 True
10.87 μs 8.48 μs 0.78 0.01 True
10.76 μs 8.33 μs 0.77 0.01 True

graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Document.Tests.Perf_EnumerateObject*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Document.Tests.Perf_EnumerateObject*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Document.Tests.Perf_EnumerateObject*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Document.Tests.Perf_EnumerateObject*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

System.Text.Json.Document.Tests.Perf_EnumerateObject.PropertyIndexer(TestCase: ObjectProperties)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 16.399385331320026 < 22.076248705912977.
IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 1008.3363607318354 (T) = (0 -16404.78565669571) / Math.Sqrt((2104.892023784441 / (125)) + (875.9729862911489 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.29431150576322246 = (23246.49727276333 - 16404.78565669571) / 23246.49727276333 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Text.Json.Document.Tests.Perf_EnumerateObject.Parse(TestCase: ObjectProperties)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 2.3792569081632653 < 2.614248737239583.
IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 151.47895709951834 (T) = (0 -2361435.770096099) / Math.Sqrt((361821769.04931957 / (124)) + (63739918.69308132 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.1259096344838039 = (2701592.2646642462 - 2361435.770096099) / 2701592.2646642462 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Text.Json.Document.Tests.Perf_EnumerateObject.PropertyIndexer(TestCase: NumericProperties)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 16.132509556433742 < 21.77010610792694.
IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 981.7548498517402 (T) = (0 -16127.3690646315) / Math.Sqrt((2596.874048789944 / (125)) + (804.3670496884918 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.2957436858012088 = (22899.85725293648 - 16127.3690646315) / 22899.85725293648 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Text.Json.Document.Tests.Perf_EnumerateObject.EnumerateProperties(TestCase: NumericProperties)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.333036124465812 < 10.210388445323016.
IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 408.26135430563625 (T) = (0 -8318.225225863027) / Math.Sqrt((1084.9542058443576 / (124)) + (813.9433416029694 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.2271996396044603 = (10763.74397859434 - 8318.225225863027) / 10763.74397859434 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Text.Json.Document.Tests.Perf_EnumerateObject.PropertyIndexer(TestCase: StringProperties)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 16.140599823451527 < 21.778962523791733.
IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 854.1649971559866 (T) = (0 -16164.935284427109) / Math.Sqrt((2210.699705605246 / (124)) + (1333.4609670433924 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.29428145817624635 = (22905.640600929746 - 16164.935284427109) / 22905.640600929746 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Text.Json.Document.Tests.Perf_EnumerateObject.EnumerateProperties(TestCase: ObjectProperties)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.479503262362638 < 10.335255269116317.
IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 451.155801104325 (T) = (0 -8471.084087227018) / Math.Sqrt((480.54284992299756 / (126)) + (736.4754029508878 / (30))) is greater than 1.975488058234171 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (126) + (30) - 2, .975) and 0.2209638944388302 = (10873.80164636268 - 8471.084087227018) / 10873.80164636268 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Text.Json.Document.Tests.Perf_EnumerateObject.EnumerateProperties(TestCase: StringProperties)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 8.329111042101173 < 10.200280006017683.
IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 207.30070435050968 (T) = (0 -8318.45812624028) / Math.Sqrt((1368.2771945766976 / (124)) + (3860.1465791054666 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.22753343956074198 = (10768.696733629537 - 8318.45812624028) / 10768.696733629537 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 736dabeca728ccf8b911d96d1b4c575b4d0db7d2
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:false, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Buffers.Text.Tests.Base64Tests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
6.82 μs 5.26 μs 0.77 0.01 True
1.30 μs 1.02 μs 0.79 0.01 True
1.30 μs 1.06 μs 0.81 0.06 True

graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Buffers.Text.Tests.Base64Tests*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Buffers.Text.Tests.Base64Tests*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Buffers.Text.Tests.Base64Tests*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Buffers.Text.Tests.Base64Tests*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

System.Buffers.Text.Tests.Base64Tests.ConvertToBase64CharArray(NumberOfBytes: 1000)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 5.259299888317599 < 6.44920028072222.
IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 413.7503059132065 (T) = (0 -5244.009906077594) / Math.Sqrt((269.9275898840811 / (125)) + (356.80214079533243 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.22825912696777992 = (6795.039746273303 - 5244.009906077594) / 6795.039746273303 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Buffers.Text.Tests.Base64Tests.Base64EncodeDestinationTooSmall(NumberOfBytes: 1000)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0231979102311317 < 1.2366388785798437.
IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 183.1772861432824 (T) = (0 -1026.66426048837) / Math.Sqrt((103.72410733402957 / (125)) + (44.51543337709748 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.21345823606295897 = (1305.2889338633386 - 1026.66426048837) / 1305.2889338633386 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Buffers.Text.Tests.Base64Tests.Base64Encode(NumberOfBytes: 1000)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0582252347775971 < 1.2369996815289257.
IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 119.86261542753721 (T) = (0 -1032.462549242332) / Math.Sqrt((285.6669608059494 / (124)) + (90.11383913578666 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.21101972888742543 = (1308.6037598714765 - 1032.462549242332) / 1308.6037598714765 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Oct 3, 2023 •
edited
Loading

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 736dabeca728ccf8b911d96d1b4c575b4d0db7d2
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:false, MonoInterpreter:false, RunKind:micro_mono

Improvements in MicroBenchmarks.Serializers.Json_FromString<LoginViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
1.53 μs 1.29 μs 0.85 0.02 True
1.54 μs 1.28 μs 0.83 0.01 True

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'MicroBenchmarks.Serializers.Json_FromString&lt;LoginViewModel&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'MicroBenchmarks.Serializers.Json_FromString&lt;LoginViewModel&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'MicroBenchmarks.Serializers.Json_FromString&lt;LoginViewModel&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'MicroBenchmarks.Serializers.Json_FromString&lt;LoginViewModel&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

MicroBenchmarks.Serializers.Json_FromString<LoginViewModel>.SystemTextJson_Reflection_

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.294243634946774 < 1.4507801921413623.
IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 78.34948749267505 (T) = (0 -1277.4090523127254) / Math.Sqrt((804.6255098290452 / (126)) + (91.94791862106442 / (30))) is greater than 1.975488058234171 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (126) + (30) - 2, .975) and 0.15864328998553848 = (1518.2728527722431 - 1277.4090523127254) / 1518.2728527722431 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

MicroBenchmarks.Serializers.Json_FromString<LoginViewModel>.SystemTextJson_SourceGen_

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.2794931267246215 < 1.4650613912718289.
IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 86.87534052499362 (T) = (0 -1282.4820516269117) / Math.Sqrt((505.0732261658756 / (125)) + (100.68094030443153 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.1555698779583103 = (1518.7545045480924 - 1282.4820516269117) / 1518.7545045480924 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 736dabeca728ccf8b911d96d1b4c575b4d0db7d2
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:false, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Tests.Perf_Uri

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
588.35 ns 488.19 ns 0.83 0.02 False
2.47 μs 456.54 ns 0.18 0.02 False
936.60 ns 819.66 ns 0.88 0.02 False
3.04 μs 2.81 μs 0.93 0.08 False
1.49 μs 1.30 μs 0.87 0.01 True
320.68 ns 300.44 ns 0.94 0.01 True
1.14 μs 956.54 ns 0.84 0.03 True
53.26 μs 3.08 μs 0.06 0.19 False
5.89 μs 1.94 μs 0.33 0.02 False
108.29 μs 39.05 μs 0.36 0.02 False
1.51 μs 1.28 μs 0.85 0.02 True
30.05 μs 24.26 μs 0.81 0.01 False
5.00 μs 1.29 μs 0.26 0.02 False
299.99 ns 273.65 ns 0.91 0.01 True
632.47 ns 528.63 ns 0.84 0.03 False
180.18 ns 168.44 ns 0.93 0.01 False
672.28 ns 614.24 ns 0.91 0.01 True
6.39 μs 2.30 μs 0.36 0.02 False
1.09 μs 915.22 ns 0.84 0.01 False
1.00 μs 870.61 ns 0.87 0.01 False
625.59 ns 556.70 ns 0.89 0.02 True
317.50 ns 300.54 ns 0.95 0.01 True

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Tests.Perf_Uri*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Tests.Perf_Uri*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Tests.Perf_Uri*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Tests.Perf_Uri*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

System.Tests.Perf_Uri.CtorIdnHostPathAndQuery(input: "http://host/")

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 488.1851559240834 < 559.8517099888167.
IsChangePoint: Marked as a change because one of 9/11/2023 3:39:38 PM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 11.275030276846762 (T) = (0 -495.55836914951067) / Math.Sqrt((1061.292804978505 / (122)) + (203.9107973615988 / (30))) is greater than 1.9759053308964478 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (122) + (30) - 2, .975) and 0.08220191337221071 = (539.9426915023447 - 495.55836914951067) / 539.9426915023447 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_Uri.BuilderToString

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 456.5387162102343 < 2.348557649987717.
IsChangePoint: Marked as a change because one of 9/11/2023 3:39:38 PM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 6.623634640962468 (T) = (0 -546.8700992864618) / Math.Sqrt((827193.6650587782 / (125)) + (30070.580506899794 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.5139194916797228 = (1125.0607459580144 - 546.8700992864618) / 1125.0607459580144 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_Uri.CtorIdnHostPathAndQuery(input: "http://dot.net/path/with?key=value#fragment")

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 819.6569053903381 < 881.6771617544462.
IsChangePoint: Marked as a change because one of 9/11/2023 8:56:48 PM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 16.060240664414508 (T) = (0 -819.4212488728461) / Math.Sqrt((899.2117510791716 / (126)) + (166.22720157797744 / (30))) is greater than 1.975488058234171 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (126) + (30) - 2, .975) and 0.0652326901050041 = (876.6045198616255 - 819.4212488728461) / 876.6045198616255 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_Uri.CtorIdnHostPathAndQuery(input: "http://höst.with.ünicode/path/with?key=value#fragment")

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 2.8131650867301294 < 2.8861623598555646.
IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 15.062499566525089 (T) = (0 -2806.1099026551656) / Math.Sqrt((4956.222902672802 / (124)) + (4098.353998991807 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.06657972607912216 = (3006.266288676121 - 2806.1099026551656) / 3006.266288676121 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_Uri.CtorIdnHostPathAndQuery(input: "https://a.much.longer.domain.name/path/with?key=value#fragment")

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.3012651950384113 < 1.4141005346415938.
IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 53.92884820983771 (T) = (0 -1301.8225517215287) / Math.Sqrt((612.0709517182945 / (125)) + (125.82827359447491 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.11103413818273429 = (1464.423559595733 - 1301.8225517215287) / 1464.423559595733 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_Uri.Ctor(input: "https://CONTOSO.com")

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 300.43877150059313 < 301.708129618612.
IsChangePoint: Marked as a change because one of 8/10/2023 8:47:51 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 46.5169052946046 (T) = (0 -300.03170734775085) / Math.Sqrt((39.575119232182246 / (125)) + (0.9352530107110498 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.08377167731595724 = (327.4639081979301 - 300.03170734775085) / 327.4639081979301 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_Uri.Ctor(input: "http://höst.with.ünicode")

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 956.5449210331877 < 1.0818394002658378.
IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 57.68658463566797 (T) = (0 -960.8637462985864) / Math.Sqrt((905.4199280991119 / (125)) + (73.61676134155253 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.1575081580030419 = (1140.502137113935 - 960.8637462985864) / 1140.502137113935 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_Uri.EscapeDataString(input: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa")

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 3.079983336742756 < 50.597315350588154.
IsChangePoint: Marked as a change because one of 9/11/2023 3:39:38 PM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 5.816044401952178 (T) = (0 -5266.5045702948) / Math.Sqrt((542343864.1059598 / (124)) + (19768135.760545425 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.7124329383037691 = (18314.00487674082 - 5266.5045702948) / 18314.00487674082 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_Uri.CtorIdnHostPathAndQuery(input: "http://host/path with escapable values?key=ünicode")

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.942923978181968 < 5.586389919068397.
IsChangePoint: Marked as a change because one of 9/11/2023 3:39:38 PM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 6.528845163900971 (T) = (0 -2109.1164307236645) / Math.Sqrt((2995364.198587243 / (123)) + (140729.42785164353 / (30))) is greater than 1.975798923817634 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (123) + (30) - 2, .975) and 0.3453553444384916 = (3221.7729310179916 - 2109.1164307236645) / 3221.7729310179916 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_Uri.EscapeDataString(input: "a{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{ü")

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 39.054338375 < 102.95538457588634.
IsChangePoint: Marked as a change because one of 9/11/2023 3:39:38 PM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 5.4860666534637454 (T) = (0 -43049.981508076125) / Math.Sqrt((1027318864.8989478 / (124)) + (62750738.54924497 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.2910319436346205 = (60722.03270874808 - 43049.981508076125) / 60722.03270874808 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_Uri.CtorIdnHostPathAndQuery(input: "http://xn--hst-sna.with.xn--nicode-2ya/path/with?key=value#fragment")

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.282218459446955 < 1.4361593344373116.
IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 52.95891925343238 (T) = (0 -1290.4311129767984) / Math.Sqrt((601.1041757488846 / (125)) + (170.745242753383 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.1173765789860924 = (1462.0404152593464 - 1290.4311129767984) / 1462.0404152593464 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_Uri.EscapeDataString(input: "{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{")

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 24.256636018826136 < 28.51202822016954.
IsChangePoint: Marked as a change because one of 9/11/2023 3:39:38 PM, 9/27/2023 6:04:31 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 9.639776532642003 (T) = (0 -24100.737879052464) / Math.Sqrt((7279543.54719199 / (132)) + (11284.088850651216 / (22))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (132) + (22) - 2, .975) and 0.08622828058050137 = (26375.009607829834 - 24100.737879052464) / 26375.009607829834 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_Uri.CtorIdnHostPathAndQuery(input: "http://host/path with escapable values?key=va lue")

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.2906940226579249 < 4.759944823659485.
IsChangePoint: Marked as a change because one of 9/11/2023 8:56:48 PM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 5.890139393396588 (T) = (0 -1482.500595035044) / Math.Sqrt((2913205.577415918 / (126)) + (132131.23428672628 / (30))) is greater than 1.975488058234171 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (126) + (30) - 2, .975) and 0.3972875448143628 = (2459.7145492512336 - 1482.500595035044) / 2459.7145492512336 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_Uri.Ctor(input: "http://dot.net")

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 273.6519234944779 < 292.7490999892628.
IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 57.557141917054935 (T) = (0 -274.1794093679212) / Math.Sqrt((19.76145476312922 / (124)) + (2.8920853263693487 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.09597719932988528 = (303.28815729501883 - 274.1794093679212) / 303.28815729501883 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_Uri.CombineAbsoluteRelative(input: "/new/path")

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 528.6296161557171 < 600.7909097774849.
IsChangePoint: Marked as a change because one of 9/11/2023 3:39:38 PM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 14.131271939489697 (T) = (0 -536.0406977703168) / Math.Sqrt((866.5634437165061 / (125)) + (193.46867793456784 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.08795334326256349 = (587.7338552918288 - 536.0406977703168) / 587.7338552918288 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_Uri.UnescapeDataString(input: "%E4%BD%A0%E5%A5%BD")

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 168.43772278547726 < 171.40010701594878.
IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 32.78093413626028 (T) = (0 -170.01586817355593) / Math.Sqrt((15.211302738224491 / (124)) + (1.601998904344341 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.07484953259732562 = (183.7710449964633 - 170.01586817355593) / 183.7710449964633 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_Uri.Ctor(input: "https://a.much.longer.domain.name")

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 614.2379351416802 < 639.591726007749.
IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 37.29306242322277 (T) = (0 -611.1639633680996) / Math.Sqrt((368.5191075023538 / (125)) + (40.933196321006086 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.11246675458520689 = (688.6096566247151 - 611.1639633680996) / 688.6096566247151 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_Uri.CtorIdnHostPathAndQuery(input: "http://host/path%20with%20escapable%20values?key=%C3%BCnicode")

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 2.3013627268346886 < 6.057663870884719.
IsChangePoint: Marked as a change because one of 9/11/2023 3:39:38 PM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 7.220119198539453 (T) = (0 -2487.8505462382245) / Math.Sqrt((3050384.756690573 / (124)) + (145583.04615899894 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.3324702855154738 = (3726.9510139475515 - 2487.8505462382245) / 3726.9510139475515 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_Uri.CtorIdnHostPathAndQuery(input: "https://CONTOSO.com/path/with?key=value#fragment")

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 915.218117765449 < 1.0367396144372396.
IsChangePoint: Marked as a change because one of 9/11/2023 3:39:38 PM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 23.067323171816472 (T) = (0 -920.879029872705) / Math.Sqrt((1502.9166868836835 / (124)) + (236.96750188239767 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.10078202746891153 = (1024.0887726928386 - 920.879029872705) / 1024.0887726928386 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_Uri.CtorIdnHostPathAndQuery(input: "https://contoso.com/path/with?key=value#fragment")

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 870.6076969015422 < 0.9548588956944178.
IsChangePoint: Marked as a change because one of 9/11/2023 3:39:38 PM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 17.76931617950986 (T) = (0 -887.1243976359387) / Math.Sqrt((699.5182989468655 / (124)) + (269.65018385932615 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.07116116070689103 = (955.0896884448578 - 887.1243976359387) / 955.0896884448578 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_Uri.Ctor(input: "http://xn--hst-sna.with.xn--nicode-2ya")

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 556.7004812326807 < 595.8590760339591.
IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 59.21050753889857 (T) = (0 -552.9485262777678) / Math.Sqrt((94.94546327540547 / (125)) + (32.39992165871821 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.126816740810131 = (633.2559865964312 - 552.9485262777678) / 633.2559865964312 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_Uri.Ctor(input: "https://contoso.com")

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 300.53887206704263 < 301.74177450494574.
IsChangePoint: Marked as a change because one of 8/10/2023 8:47:51 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 44.44654050601596 (T) = (0 -300.0710944179189) / Math.Sqrt((42.96845884328328 / (124)) + (1.0979038685776725 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.08398148541728737 = (327.5819097986406 - 300.0710944179189) / 327.5819097986406 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Oct 3, 2023 •
edited
Loading

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 736dabeca728ccf8b911d96d1b4c575b4d0db7d2
Compare fc5c29692fc1a92426b7d1ce8c501e7696062bb6
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:false, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.IO.Tests.BinaryReaderTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
28.43 ns 25.12 ns 0.88 0.01 True
28.31 ns 25.13 ns 0.89 0.01 True
30.49 ns 25.01 ns 0.82 0.01 True
32.03 ns 26.95 ns 0.84 0.01 True
178.09 ns 156.03 ns 0.88 0.01 True
29.20 ns 25.20 ns 0.86 0.01 True
29.41 ns 24.76 ns 0.84 0.01 True
3.65 μs 3.07 μs 0.84 0.01 False

graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.IO.Tests.BinaryReaderTests*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.IO.Tests.BinaryReaderTests*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.IO.Tests.BinaryReaderTests*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.IO.Tests.BinaryReaderTests*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

System.IO.Tests.BinaryReaderTests.ReadUInt32

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 25.117893093318973 < 26.96627119333546.
IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 169.84705765989563 (T) = (0 -25.014112830879178) / Math.Sqrt((0.01086485790923212 / (125)) + (0.00868764125312273 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.11641471296060346 = (28.309788763791254 - 25.014112830879178) / 28.309788763791254 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.IO.Tests.BinaryReaderTests.ReadUInt16

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 25.12630301618481 < 26.87376959679426.
IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 154.2804872266557 (T) = (0 -25.15187201579009) / Math.Sqrt((0.012149456293956126 / (124)) + (0.009476773039310283 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.11094376513247053 = (28.29052992304559 - 25.15187201579009) / 28.29052992304559 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.IO.Tests.BinaryReaderTests.ReadSingle

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 25.014134052856086 < 29.057432910822442.
IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 222.72796165485312 (T) = (0 -25.082840735676562) / Math.Sqrt((0.017473369895031024 / (126)) + (0.014649052768618685 / (30))) is greater than 1.975488058234171 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (126) + (30) - 2, .975) and 0.1818994230082375 = (30.659849706876717 - 25.082840735676562) / 30.659849706876717 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.IO.Tests.BinaryReaderTests.ReadHalf

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 26.95460968734937 < 30.510393384984205.
IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 260.99157689020063 (T) = (0 -26.929015998567095) / Math.Sqrt((0.014826918606784367 / (125)) + (0.008010857020571758 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.15989436141282568 = (32.054321220667276 - 26.929015998567095) / 32.054321220667276 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.IO.Tests.BinaryReaderTests.ReadSmallString

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 156.03216179399047 < 169.2905401780105.
IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 72.01578489339548 (T) = (0 -155.37143776182515) / Math.Sqrt((4.555556600746096 / (124)) + (1.2749648739011918 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.11541461404506284 = (175.64323380054137 - 155.37143776182515) / 175.64323380054137 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.IO.Tests.BinaryReaderTests.ReadUInt64

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 25.20284677332841 < 27.750168263540296.
IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 154.82390760902183 (T) = (0 -25.114738394668446) / Math.Sqrt((0.03642275468837057 / (123)) + (0.011809845883022176 / (30))) is greater than 1.975798923817634 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (123) + (30) - 2, .975) and 0.13934578392218555 = (29.18098572632536 - 25.114738394668446) / 29.18098572632536 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.IO.Tests.BinaryReaderTests.ReadDouble

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 24.762295834434216 < 27.91188225734174.
IsChangePoint: Marked as a change because one of 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 200.98850114882976 (T) = (0 -25.074594496239488) / Math.Sqrt((0.016890893822289928 / (125)) + (0.009574517314504758 / (30))) is greater than 1.9755903150050484 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (125) + (30) - 2, .975) and 0.1459147620002954 = (29.358421596145373 - 25.074594496239488) / 29.358421596145373 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

System.IO.Tests.BinaryReaderTests.ReadLargeString

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 3.072559414194704 < 3.4831941459056863.
IsChangePoint: Marked as a change because one of 8/9/2023 9:46:11 AM, 9/25/2023 10:28:34 PM, 10/2/2023 10:24:51 PM falls between 9/20/2023 11:58:56 PM and 10/2/2023 10:24:51 PM.
IsImprovementStdDev: Marked as improvement because 35.228990127095834 (T) = (0 -3088.5181119973886) / Math.Sqrt((37917.47233309074 / (124)) + (170.57276459412543 / (30))) is greater than 1.975693927815077 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (124) + (30) - 2, .975) and 0.16757349442294478 = (3710.2592136423673 - 3088.5181119973886) / 3710.2592136423673 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@matouskozak
Copy link
Member

Probably caused by dotnet/runtime@169e22c.

@matouskozak matouskozak mentioned this issue Oct 16, 2023
Closed
This was referenced Oct 16, 2023
Closed
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
arch-x64 branch-refs/heads/main kind-micro_mono mono-jit os-linux perf-improvement PGO Applied if there were any profile guided optimization updates in the observed interval. runtime-mono
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@matouskozak