Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Linux/arm64: 48 Regressions on 4/17/2023 5:51:51 AM #16810

Closed
performanceautofiler bot opened this issue Apr 20, 2023 · 3 comments
Closed

[Perf] Linux/arm64: 48 Regressions on 4/17/2023 5:51:51 AM #16810

performanceautofiler bot opened this issue Apr 20, 2023 · 3 comments
Assignees
@EgorBo
Labels
ampere arch-arm64 branch-refs/heads/main kind-micro os-linux perf-regression runtime-coreclr untriaged

Comments

@performanceautofiler
Copy link

performanceautofiler bot commented Apr 20, 2023 •
edited
Loading

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 0b081118dedcbc01d445f3f56222fde9d86f3f44
Compare e3f55cbfd83162bc85d73d69c765e7f44c478e6d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Json.Tests.Perf_Strings

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
WriteStringsUtf8 - Duration of single invocation 71.92 ms 79.03 ms 1.10 0.01 True
WriteStringsUtf8 - Duration of single invocation 72.58 ms 79.58 ms 1.10 0.00 True
WriteStringsUtf8 - Duration of single invocation 72.60 ms 79.77 ms 1.10 0.00 True
WriteStringsUtf8 - Duration of single invocation 72.56 ms 79.36 ms 1.09 0.00 True

graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Tests.Perf_Strings*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Tests.Perf_Strings.WriteStringsUtf8(Formatted: False, SkipValidation: False, Escaped: AllEscaped)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 79.02785479166667 > 75.59320740000003.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/20/2023 3:04:43 PM falls between 4/11/2023 11:54:05 PM and 4/20/2023 3:04:43 PM.
IsRegressionStdDev: Marked as regression because -11.078555428249551 (T) = (0 -79367520.27494505) / Math.Sqrt((3976815729587.9863 / (13)) + (66875034913.446014 / (10))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (10) - 2, .025) and -0.08464893692354053 = (73173464.31008382 - 79367520.27494505) / 73173464.31008382 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Text.Json.Tests.Perf_Strings.WriteStringsUtf8(Formatted: True, SkipValidation: True, Escaped: AllEscaped)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 79.58466659999999 > 76.48231129831731.
IsChangePoint: Marked as a change because one of 4/12/2023 2:59:13 PM, 4/20/2023 3:04:43 PM falls between 4/11/2023 11:54:05 PM and 4/20/2023 3:04:43 PM.
IsRegressionStdDev: Marked as regression because -26.660182322065232 (T) = (0 -79745454.98382033) / Math.Sqrt((679847484189.8573 / (12)) + (19797634626.365692 / (11))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (11) - 2, .025) and -0.08793599452982431 = (73299767.06790008 - 79745454.98382033) / 73299767.06790008 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Text.Json.Tests.Perf_Strings.WriteStringsUtf8(Formatted: True, SkipValidation: False, Escaped: AllEscaped)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 79.76996126666667 > 76.23633796086538.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/20/2023 3:04:43 PM falls between 4/11/2023 11:54:05 PM and 4/20/2023 3:04:43 PM.
IsRegressionStdDev: Marked as regression because -19.667059750047258 (T) = (0 -79885958.30322343) / Math.Sqrt((1345269588644.221 / (13)) + (95090992402.67035 / (10))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (10) - 2, .025) and -0.09022066979254205 = (73275035.51957506 - 79885958.30322343) / 73275035.51957506 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Text.Json.Tests.Perf_Strings.WriteStringsUtf8(Formatted: False, SkipValidation: True, Escaped: AllEscaped)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 79.36374364285714 > 75.75851303124999.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/20/2023 3:04:43 PM falls between 4/11/2023 11:54:05 PM and 4/20/2023 3:04:43 PM.
IsRegressionStdDev: Marked as regression because -31.635729818532084 (T) = (0 -79106519.4847436) / Math.Sqrt((437905611916.5664 / (13)) + (107021974989.30809 / (10))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (10) - 2, .025) and -0.09200682095031451 = (72441415.16982602 - 79106519.4847436) / 72441415.16982602 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 0b081118dedcbc01d445f3f56222fde9d86f3f44
Compare e3f55cbfd83162bc85d73d69c765e7f44c478e6d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Json.Tests.Perf_Deep

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
WriteDeepUtf8 - Duration of single invocation 5.86 ms 6.28 ms 1.07 0.00 False
WriteDeepUtf16 - Duration of single invocation 5.95 ms 6.35 ms 1.07 0.01 False
WriteDeepUtf8 - Duration of single invocation 5.48 ms 6.03 ms 1.10 0.01 False
WriteDeepUtf16 - Duration of single invocation 5.55 ms 6.09 ms 1.10 0.00 False

graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Tests.Perf_Deep*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Tests.Perf_Deep.WriteDeepUtf8(Formatted: False, SkipValidation: False)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 6.279604376725839 > 6.145239236666667.
IsChangePoint: Marked as a change because one of 1/17/2023 4:06:07 PM, 4/12/2023 8:39:35 PM, 4/20/2023 3:04:43 PM falls between 4/11/2023 11:54:05 PM and 4/20/2023 3:04:43 PM.
IsRegressionStdDev: Marked as regression because -30.08089794867182 (T) = (0 -6377517.11258584) / Math.Sqrt((1040258228.1073698 / (13)) + (2690849982.878503 / (11))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (11) - 2, .025) and -0.09287817276846606 = (5835524.280286785 - 6377517.11258584) / 5835524.280286785 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.Json.Tests.Perf_Deep.WriteDeepUtf16(Formatted: False, SkipValidation: False)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 6.347093615789475 > 6.1724763356003765.
IsChangePoint: Marked as a change because one of 1/17/2023 4:06:07 PM, 4/12/2023 8:39:35 PM, 4/20/2023 3:04:43 PM falls between 4/11/2023 11:54:05 PM and 4/20/2023 3:04:43 PM.
IsRegressionStdDev: Marked as regression because -30.550681714008185 (T) = (0 -6374113.400817953) / Math.Sqrt((903655909.3945886 / (13)) + (1931651174.6346579 / (10))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (10) - 2, .025) and -0.0842230303153939 = (5878968.830761474 - 6374113.400817953) / 5878968.830761474 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.Json.Tests.Perf_Deep.WriteDeepUtf8(Formatted: False, SkipValidation: True)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 6.029870440650407 > 5.8175997940668704.
IsChangePoint: Marked as a change because one of 1/17/2023 4:06:07 PM, 4/12/2023 8:39:35 PM, 4/20/2023 3:04:43 PM falls between 4/11/2023 11:54:05 PM and 4/20/2023 3:04:43 PM.
IsRegressionStdDev: Marked as regression because -42.55451658598175 (T) = (0 -6063355.365597437) / Math.Sqrt((941985076.0975834 / (13)) + (868646562.46448 / (10))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (10) - 2, .025) and -0.09719857577142185 = (5526215.125948732 - 6063355.365597437) / 5526215.125948732 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.Json.Tests.Perf_Deep.WriteDeepUtf16(Formatted: False, SkipValidation: True)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 6.08501673939394 > 5.827692158803029.
IsChangePoint: Marked as a change because one of 1/17/2023 4:06:07 PM, 4/12/2023 8:39:35 PM, 4/20/2023 3:04:43 PM falls between 4/11/2023 11:54:05 PM and 4/20/2023 3:04:43 PM.
IsRegressionStdDev: Marked as regression because -73.33598056016277 (T) = (0 -6077643.223764513) / Math.Sqrt((38494989.23867301 / (13)) + (527614397.62802714 / (11))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (11) - 2, .025) and -0.0942232408088191 = (5554299.156790062 - 6077643.223764513) / 5554299.156790062 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 0b081118dedcbc01d445f3f56222fde9d86f3f44
Compare e3f55cbfd83162bc85d73d69c765e7f44c478e6d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Json.Tests.Perf_Basic

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
WriteBasicUtf8 - Duration of single invocation 955.47 ns 1.03 μs 1.08 0.00 False
WriteBasicUtf16 - Duration of single invocation 823.89 ns 900.16 ns 1.09 0.01 False
WriteBasicUtf8 - Duration of single invocation 3.12 ms 3.55 ms 1.14 0.00 True
WriteBasicUtf8 - Duration of single invocation 2.29 ms 2.73 ms 1.19 0.00 True
WriteBasicUtf16 - Duration of single invocation 1.01 μs 1.08 μs 1.07 0.00 False
WriteBasicUtf16 - Duration of single invocation 2.97 ms 3.38 ms 1.14 0.00 True
WriteBasicUtf16 - Duration of single invocation 3.12 ms 3.54 ms 1.14 0.00 True
WriteBasicUtf16 - Duration of single invocation 2.44 ms 2.86 ms 1.17 0.00 True
WriteBasicUtf8 - Duration of single invocation 2.97 ms 3.39 ms 1.14 0.00 True
WriteBasicUtf8 - Duration of single invocation 914.90 ns 973.49 ns 1.06 0.00 False
WriteBasicUtf16 - Duration of single invocation 891.20 ns 957.64 ns 1.07 0.00 False
WriteBasicUtf16 - Duration of single invocation 2.30 ms 2.73 ms 1.19 0.00 True
WriteBasicUtf8 - Duration of single invocation 727.46 ns 792.01 ns 1.09 0.01 False
WriteBasicUtf8 - Duration of single invocation 797.71 ns 846.44 ns 1.06 0.00 False
WriteBasicUtf16 - Duration of single invocation 1.05 μs 1.13 μs 1.07 0.01 False
WriteBasicUtf8 - Duration of single invocation 2.44 ms 2.86 ms 1.17 0.00 True

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Tests.Perf_Basic*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted: True, SkipValidation: False, DataSize: 10)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.029871698432798 > 1005.9538409099137.
IsChangePoint: Marked as a change because one of 1/19/2023 8:10:32 PM, 4/12/2023 8:39:35 PM, 4/20/2023 3:04:43 PM falls between 4/11/2023 11:54:05 PM and 4/20/2023 3:04:43 PM.
IsRegressionStdDev: Marked as regression because -28.235613970741696 (T) = (0 -1028.5638252818128) / Math.Sqrt((18.659705699767475 / (13)) + (52.88227409585289 / (10))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (10) - 2, .025) and -0.07663650342510717 = (955.3492028271746 - 1028.5638252818128) / 955.3492028271746 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: False, SkipValidation: True, DataSize: 10)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 900.1571260101153 > 865.0697649651012.
IsChangePoint: Marked as a change because one of 1/21/2023 1:51:00 AM, 4/12/2023 8:39:35 PM, 4/20/2023 3:04:43 PM falls between 4/11/2023 11:54:05 PM and 4/20/2023 3:04:43 PM.
IsRegressionStdDev: Marked as regression because -101.37945101584813 (T) = (0 -898.9262485676984) / Math.Sqrt((1.788218719959587 / (13)) + (4.619488065127694 / (11))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (11) - 2, .025) and -0.09195063227880435 = (823.2297523302119 - 898.9262485676984) / 823.2297523302119 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted: True, SkipValidation: False, DataSize: 100000)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.547405436734694 > 3.2778566450000004.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/20/2023 3:04:43 PM falls between 4/11/2023 11:54:05 PM and 4/20/2023 3:04:43 PM.
IsRegressionStdDev: Marked as regression because -47.777144372428 (T) = (0 -3528746.8200437548) / Math.Sqrt((143764962.0671961 / (13)) + (669331115.706651 / (11))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (11) - 2, .025) and -0.1297029718193374 = (3123605.857529844 - 3528746.8200437548) / 3123605.857529844 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted: False, SkipValidation: True, DataSize: 100000)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.734424858608059 > 2.4079806608796295.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/20/2023 3:04:43 PM falls between 4/11/2023 11:54:05 PM and 4/20/2023 3:04:43 PM.
IsRegressionStdDev: Marked as regression because -179.58998892639426 (T) = (0 -2721282.8835088825) / Math.Sqrt((2678227.864333186 / (13)) + (60172929.909566306 / (11))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (11) - 2, .025) and -0.18656594234016807 = (2293410.577874767 - 2721282.8835088825) / 2293410.577874767 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: True, SkipValidation: True, DataSize: 10)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.082039624705947 > 1.058997764280184.
IsChangePoint: Marked as a change because one of 1/17/2023 4:06:07 PM, 4/12/2023 8:39:35 PM, 4/20/2023 3:04:43 PM falls between 4/11/2023 11:54:05 PM and 4/20/2023 3:04:43 PM.
IsRegressionStdDev: Marked as regression because -30.974579480954418 (T) = (0 -1079.5181181289472) / Math.Sqrt((1.3383966970722276 / (13)) + (56.48945237717628 / (11))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (11) - 2, .025) and -0.07028671056600025 = (1008.6251725559267 - 1079.5181181289472) / 1008.6251725559267 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: True, SkipValidation: True, DataSize: 100000)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.3781252794520547 > 3.116194016039157.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/20/2023 3:04:43 PM falls between 4/11/2023 11:54:05 PM and 4/20/2023 3:04:43 PM.
IsRegressionStdDev: Marked as regression because -61.5568009092439 (T) = (0 -3368059.4923759294) / Math.Sqrt((74901915.30176198 / (13)) + (394360874.2066844 / (11))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (11) - 2, .025) and -0.13365667429021538 = (2970969.5790259233 - 3368059.4923759294) / 2970969.5790259233 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: True, SkipValidation: False, DataSize: 100000)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.5440747806122452 > 3.2756833547784816.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/20/2023 3:04:43 PM falls between 4/11/2023 11:54:05 PM and 4/20/2023 3:04:43 PM.
IsRegressionStdDev: Marked as regression because -38.739757273689285 (T) = (0 -3534021.271883682) / Math.Sqrt((6605911.532220308 / (12)) + (1249175834.544349 / (11))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (11) - 2, .025) and -0.13262984657393598 = (3120190.839552441 - 3534021.271883682) / 3120190.839552441 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: False, SkipValidation: False, DataSize: 100000)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.856123393103448 > 2.5611899108823533.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/20/2023 3:04:43 PM falls between 4/11/2023 11:54:05 PM and 4/20/2023 3:04:43 PM.
IsRegressionStdDev: Marked as regression because -227.10794800158283 (T) = (0 -2859117.10044129) / Math.Sqrt((948269.1683552613 / (13)) + (36952277.84652247 / (11))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (11) - 2, .025) and -0.1725526711585291 = (2438369.866674192 - 2859117.10044129) / 2438369.866674192 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted: True, SkipValidation: True, DataSize: 100000)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.386529800913242 > 3.1205535108433744.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/20/2023 3:04:43 PM falls between 4/11/2023 11:54:05 PM and 4/20/2023 3:04:43 PM.
IsRegressionStdDev: Marked as regression because -58.85943842568651 (T) = (0 -3370486.582995267) / Math.Sqrt((7812633.704994358 / (13)) + (500751187.9102931 / (11))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (11) - 2, .025) and -0.13455914742102934 = (2970745.5892949547 - 3370486.582995267) / 2970745.5892949547 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted: True, SkipValidation: True, DataSize: 10)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 973.489355510868 > 960.9089581123062.
IsChangePoint: Marked as a change because one of 1/19/2023 8:10:32 PM, 4/12/2023 8:39:35 PM, 4/20/2023 3:04:43 PM falls between 4/11/2023 11:54:05 PM and 4/20/2023 3:04:43 PM.
IsRegressionStdDev: Marked as regression because -29.39374267282879 (T) = (0 -975.076519186396) / Math.Sqrt((37.411070759077866 / (13)) + (19.1593647499301 / (10))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (10) - 2, .025) and -0.07066523741578008 = (910.7202560717274 - 975.076519186396) / 910.7202560717274 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: False, SkipValidation: False, DataSize: 10)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 957.6360073147652 > 935.7524904893553.
IsChangePoint: Marked as a change because one of 1/17/2023 4:06:07 PM, 4/12/2023 8:39:35 PM, 4/20/2023 3:04:43 PM falls between 4/11/2023 11:54:05 PM and 4/20/2023 3:04:43 PM.
IsRegressionStdDev: Marked as regression because -59.95832090620941 (T) = (0 -956.2053339415663) / Math.Sqrt((8.288642068242225 / (13)) + (5.456445761393925 / (11))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (11) - 2, .025) and -0.07153878315102594 = (892.3665190444123 - 956.2053339415663) / 892.3665190444123 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: False, SkipValidation: True, DataSize: 100000)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.726388371428571 > 2.412493137152778.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/20/2023 3:04:43 PM falls between 4/11/2023 11:54:05 PM and 4/20/2023 3:04:43 PM.
IsRegressionStdDev: Marked as regression because -172.19929811905013 (T) = (0 -2717853.710614748) / Math.Sqrt((688065.6690407739 / (13)) + (65022006.22972976 / (11))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (11) - 2, .025) and -0.18305408781076504 = (2297319.9100677813 - 2717853.710614748) / 2297319.9100677813 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted: False, SkipValidation: True, DataSize: 10)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 792.0111705359393 > 767.7834185541697.
IsChangePoint: Marked as a change because one of 1/19/2023 8:10:32 PM, 4/12/2023 8:39:35 PM, 4/20/2023 3:04:43 PM falls between 4/11/2023 11:54:05 PM and 4/20/2023 3:04:43 PM.
IsRegressionStdDev: Marked as regression because -29.24495891159728 (T) = (0 -791.4010149065307) / Math.Sqrt((61.451920175212585 / (13)) + (2.8288915527699254 / (11))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (11) - 2, .025) and -0.08991826891785117 = (726.1104226579212 - 791.4010149065307) / 726.1104226579212 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted: False, SkipValidation: False, DataSize: 10)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 846.438580431851 > 838.5006715303779.
IsChangePoint: Marked as a change because one of 1/19/2023 8:10:32 PM, 4/12/2023 8:39:35 PM, 4/20/2023 3:04:43 PM falls between 4/11/2023 11:54:05 PM and 4/20/2023 3:04:43 PM.
IsRegressionStdDev: Marked as regression because -23.924305857566022 (T) = (0 -848.2317868747541) / Math.Sqrt((52.8929056417416 / (13)) + (12.80789248734005 / (11))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (11) - 2, .025) and -0.06897120122934113 = (793.5029361869322 - 848.2317868747541) / 793.5029361869322 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: True, SkipValidation: False, DataSize: 10)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.1303580627069272 > 1.108388953497812.
IsChangePoint: Marked as a change because one of 1/21/2023 1:51:00 AM, 4/12/2023 8:39:35 PM, 4/20/2023 3:04:43 PM falls between 4/11/2023 11:54:05 PM and 4/20/2023 3:04:43 PM.
IsRegressionStdDev: Marked as regression because -41.21657205827034 (T) = (0 -1133.0078304123892) / Math.Sqrt((2.830477238600441 / (13)) + (35.86251579552429 / (11))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (11) - 2, .025) and -0.07277995043846416 = (1056.1418769518473 - 1133.0078304123892) / 1056.1418769518473 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted: False, SkipValidation: False, DataSize: 100000)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.856320570881226 > 2.5594023736887257.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/20/2023 3:04:43 PM falls between 4/11/2023 11:54:05 PM and 4/20/2023 3:04:43 PM.
IsRegressionStdDev: Marked as regression because -226.04015955501245 (T) = (0 -2854614.7032858543) / Math.Sqrt((4277806.43515459 / (13)) + (30634195.815776553 / (10))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (10) - 2, .025) and -0.1707501437398091 = (2438278.328258119 - 2854614.7032858543) / 2438278.328258119 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler
Copy link
Author

performanceautofiler bot commented Apr 20, 2023 •
edited
Loading

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 0b081118dedcbc01d445f3f56222fde9d86f3f44
Compare e3f55cbfd83162bc85d73d69c765e7f44c478e6d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Json.Serialization.Tests.WriteJson<LargeStructWithProperties>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SerializeToWriter - Duration of single invocation 379.28 ns 414.13 ns 1.09 0.02 False
SerializeToUtf8Bytes - Duration of single invocation 474.83 ns 512.72 ns 1.08 0.03 False
SerializeToString - Duration of single invocation 515.01 ns 552.70 ns 1.07 0.03 False

graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;LargeStructWithProperties&gt;*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.WriteJson<LargeStructWithProperties>.SerializeToWriter(Mode: SourceGen)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 414.13298479762733 > 398.0745708262128.
IsChangePoint: Marked as a change because one of 1/13/2023 9:23:19 AM, 1/19/2023 8:10:32 PM, 4/12/2023 8:39:35 PM, 4/20/2023 3:04:43 PM falls between 4/11/2023 11:54:05 PM and 4/20/2023 3:04:43 PM.
IsRegressionStdDev: Marked as regression because -11.650138522761917 (T) = (0 -409.04481549455505) / Math.Sqrt((116.25379923280757 / (13)) + (10.101430604209119 / (11))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (11) - 2, .025) and -0.09822202499411212 = (372.4609470446088 - 409.04481549455505) / 372.4609470446088 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.Json.Serialization.Tests.WriteJson<LargeStructWithProperties>.SerializeToUtf8Bytes(Mode: SourceGen)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 512.7220573692568 > 501.71138391094445.
IsChangePoint: Marked as a change because one of 1/20/2023 5:04:01 PM, 2/24/2023 4:59:28 PM, 4/12/2023 8:39:35 PM, 4/20/2023 3:04:43 PM falls between 4/11/2023 11:54:05 PM and 4/20/2023 3:04:43 PM.
IsRegressionStdDev: Marked as regression because -11.604255801564026 (T) = (0 -511.8125885413339) / Math.Sqrt((117.73626876832874 / (13)) + (31.353543324700475 / (10))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (10) - 2, .025) and -0.08597312515578123 = (471.2939728300506 - 511.8125885413339) / 471.2939728300506 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.Json.Serialization.Tests.WriteJson<LargeStructWithProperties>.SerializeToString(Mode: SourceGen)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 552.7014381847966 > 539.3841293413229.
IsChangePoint: Marked as a change because one of 1/20/2023 1:51:38 AM, 4/12/2023 8:39:35 PM, 4/20/2023 3:04:43 PM falls between 4/11/2023 11:54:05 PM and 4/20/2023 3:04:43 PM.
IsRegressionStdDev: Marked as regression because -12.657800943404173 (T) = (0 -549.6464739550577) / Math.Sqrt((118.2197613327131 / (13)) + (19.686307552035014 / (11))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (11) - 2, .025) and -0.08221939193899977 = (507.8882138401369 - 549.6464739550577) / 507.8882138401369 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 0b081118dedcbc01d445f3f56222fde9d86f3f44
Compare e3f55cbfd83162bc85d73d69c765e7f44c478e6d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Memory.Span<Char>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SequenceCompareTo - Duration of single invocation 6.71 ns 8.09 ns 1.21 0.06 False
LastIndexOfValue - Duration of single invocation 27.49 ns 30.87 ns 1.12 0.09 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.Span&lt;Char&gt;*'

Payloads

Baseline
Compare

Histogram

System.Memory.Span<Char>.SequenceCompareTo(Size: 33)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 8.088887383217697 > 7.0449684658208405.
IsChangePoint: Marked as a change because one of 2/14/2023 11:21:41 AM, 3/3/2023 4:19:26 AM, 4/3/2023 11:26:06 PM, 4/6/2023 11:23:56 PM, 4/12/2023 8:39:35 PM, 4/20/2023 3:04:43 PM falls between 4/11/2023 11:54:05 PM and 4/20/2023 3:04:43 PM.
IsRegressionStdDev: Marked as regression because -8.810188718447675 (T) = (0 -8.06890040009743) / Math.Sqrt((0.2286819351561746 / (13)) + (0.027913717959557643 / (10))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (10) - 2, .025) and -0.18466918653040149 = (6.811100087552047 - 8.06890040009743) / 6.811100087552047 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Memory.Span<Char>.LastIndexOfValue(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 30.87158028202543 > 29.015006344695994.
IsChangePoint: Marked as a change because one of 1/5/2023 10:03:03 PM, 2/7/2023 2:48:42 AM, 2/15/2023 12:21:17 PM, 3/3/2023 4:19:26 AM, 4/6/2023 11:23:56 PM, 4/12/2023 8:39:35 PM, 4/20/2023 3:04:43 PM falls between 4/11/2023 11:54:05 PM and 4/20/2023 3:04:43 PM.
IsRegressionStdDev: Marked as regression because -43.93349770078966 (T) = (0 -30.651457410657212) / Math.Sqrt((0.011958167218238816 / (13)) + (0.037831768353039724 / (10))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (10) - 2, .025) and -0.10901081243310431 = (27.63855596990053 - 30.651457410657212) / 27.63855596990053 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 0b081118dedcbc01d445f3f56222fde9d86f3f44
Compare e3f55cbfd83162bc85d73d69c765e7f44c478e6d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Globalization.Tests.Perf_NumberCultureInfo

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ToString - Duration of single invocation 146.09 ns 158.89 ns 1.09 0.00 False
ToString - Duration of single invocation 146.36 ns 158.14 ns 1.08 0.00 False
ToString - Duration of single invocation 147.22 ns 158.08 ns 1.07 0.00 False
ToString - Duration of single invocation 146.84 ns 157.98 ns 1.08 0.00 False

graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Globalization.Tests.Perf_NumberCultureInfo*'

Payloads

Baseline
Compare

Histogram

System.Globalization.Tests.Perf_NumberCultureInfo.ToString(culturestring: da)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 158.89277160705487 > 154.0834847134949.
IsChangePoint: Marked as a change because one of 2/17/2023 12:28:21 PM, 3/20/2023 8:31:51 AM, 4/6/2023 1:55:21 PM, 4/12/2023 8:39:35 PM, 4/20/2023 3:04:43 PM falls between 4/11/2023 11:54:05 PM and 4/20/2023 3:04:43 PM.
IsRegressionStdDev: Marked as regression because -21.92554760990944 (T) = (0 -158.85866136628917) / Math.Sqrt((1.2957381702381114 / (12)) + (1.7028441471090037 / (11))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (11) - 2, .025) and -0.07613875851615219 = (147.6191244940694 - 158.85866136628917) / 147.6191244940694 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Globalization.Tests.Perf_NumberCultureInfo.ToString(culturestring: fr)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 158.14282654152126 > 154.21149100853222.
IsChangePoint: Marked as a change because one of 2/17/2023 12:28:21 PM, 3/18/2023 3:40:27 PM, 4/6/2023 1:55:21 PM, 4/12/2023 8:39:35 PM, 4/20/2023 3:04:43 PM falls between 4/11/2023 11:54:05 PM and 4/20/2023 3:04:43 PM.
IsRegressionStdDev: Marked as regression because -39.48066556781831 (T) = (0 -158.1327552175636) / Math.Sqrt((0.6111874996603371 / (13)) + (0.34481286627936847 / (11))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (11) - 2, .025) and -0.07514122258519987 = (147.08091541437696 - 158.1327552175636) / 147.08091541437696 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Globalization.Tests.Perf_NumberCultureInfo.ToString(culturestring: ja)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 158.07780246297438 > 154.42514917975322.
IsChangePoint: Marked as a change because one of 2/17/2023 12:28:21 PM, 3/18/2023 12:49:26 AM, 4/6/2023 1:55:21 PM, 4/12/2023 8:39:35 PM, 4/20/2023 3:04:43 PM falls between 4/11/2023 11:54:05 PM and 4/20/2023 3:04:43 PM.
IsRegressionStdDev: Marked as regression because -39.87081100222101 (T) = (0 -158.3159586442586) / Math.Sqrt((0.6129019068432151 / (13)) + (0.3303862116871414 / (11))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (11) - 2, .025) and -0.0752294490514308 = (147.23923231820444 - 158.3159586442586) / 147.23923231820444 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Globalization.Tests.Perf_NumberCultureInfo.ToString(culturestring: )

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 157.98462234373625 > 153.85920075162292.
IsChangePoint: Marked as a change because one of 2/17/2023 12:28:21 PM, 3/17/2023 1:30:52 PM, 4/6/2023 1:55:21 PM, 4/12/2023 8:39:35 PM, 4/20/2023 3:04:43 PM falls between 4/11/2023 11:54:05 PM and 4/20/2023 3:04:43 PM.
IsRegressionStdDev: Marked as regression because -35.96547821995643 (T) = (0 -158.0261440202247) / Math.Sqrt((0.8150969700745089 / (13)) + (0.322769232497326 / (11))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (11) - 2, .025) and -0.07416920402825807 = (147.11475941370176 - 158.0261440202247) / 147.11475941370176 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 0b081118dedcbc01d445f3f56222fde9d86f3f44
Compare e3f55cbfd83162bc85d73d69c765e7f44c478e6d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_Int64

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
TryFormat - Duration of single invocation 34.90 ns 38.78 ns 1.11 0.01 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Int64*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Int64.TryFormat(value: -9223372036854775808)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 38.783564728493495 > 37.38500995252504.
IsChangePoint: Marked as a change because one of 1/21/2023 1:51:00 AM, 4/12/2023 8:39:35 PM, 4/20/2023 3:04:43 PM falls between 4/11/2023 11:54:05 PM and 4/20/2023 3:04:43 PM.
IsRegressionStdDev: Marked as regression because -24.806933520389098 (T) = (0 -38.51365959571221) / Math.Sqrt((0.0847663934745251 / (13)) + (0.06460451374661615 / (10))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (10) - 2, .025) and -0.07919767273922317 = (35.68730786636772 - 38.51365959571221) / 35.68730786636772 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 0b081118dedcbc01d445f3f56222fde9d86f3f44
Compare e3f55cbfd83162bc85d73d69c765e7f44c478e6d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IsMatch - Duration of single invocation 231.42 ns 248.40 ns 1.07 0.11 False
IsMatch - Duration of single invocation 233.26 ns 251.71 ns 1.08 0.09 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple*'

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 13, Options: NonBacktracking)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 248.40453007444492 > 240.96870852045245.
IsChangePoint: Marked as a change because one of 1/9/2023 10:41:51 AM, 3/1/2023 5:53:50 PM, 4/12/2023 8:39:35 PM, 4/20/2023 3:04:43 PM falls between 4/11/2023 11:54:05 PM and 4/20/2023 3:04:43 PM.
IsRegressionStdDev: Marked as regression because -7.144888208698571 (T) = (0 -247.78233506165998) / Math.Sqrt((30.267817600143413 / (13)) + (22.546493425990704 / (10))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (10) - 2, .025) and -0.0657914659276726 = (232.48669461430566 - 247.78233506165998) / 232.48669461430566 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 12, Options: NonBacktracking)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 251.7120807454962 > 243.9901297512286.
IsChangePoint: Marked as a change because one of 3/5/2023 5:50:49 PM, 4/12/2023 8:39:35 PM, 4/20/2023 3:04:43 PM falls between 4/11/2023 11:54:05 PM and 4/20/2023 3:04:43 PM.
IsRegressionStdDev: Marked as regression because -8.092126946391001 (T) = (0 -248.7500452553129) / Math.Sqrt((23.112645138360932 / (13)) + (17.795323918821925 / (10))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (10) - 2, .025) and -0.06536832699631977 = (233.48736671816994 - 248.7500452553129) / 233.48736671816994 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 0b081118dedcbc01d445f3f56222fde9d86f3f44
Compare e3f55cbfd83162bc85d73d69c765e7f44c478e6d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in Microsoft.Extensions.Primitives.StringSegmentBenchmark

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
GetSegmentHashCode - Duration of single invocation 13.20 ns 15.04 ns 1.14 0.01 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Microsoft.Extensions.Primitives.StringSegmentBenchmark*'

Payloads

Baseline
Compare

Histogram

Microsoft.Extensions.Primitives.StringSegmentBenchmark.GetSegmentHashCode

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.035221018060556 > 13.855504675343484.
IsChangePoint: Marked as a change because one of 2/14/2023 11:21:41 AM, 3/3/2023 4:19:26 AM, 4/3/2023 11:26:06 PM, 4/6/2023 11:23:56 PM, 4/12/2023 8:39:35 PM, 4/20/2023 3:04:43 PM falls between 4/11/2023 11:54:05 PM and 4/20/2023 3:04:43 PM.
IsRegressionStdDev: Marked as regression because -47.364535940483286 (T) = (0 -14.989370333770745) / Math.Sqrt((0.009250366083992653 / (13)) + (0.00874894673308613 / (11))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (11) - 2, .025) and -0.1398136705040098 = (13.150719912968453 - 14.989370333770745) / 13.150719912968453 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 0b081118dedcbc01d445f3f56222fde9d86f3f44
Compare e3f55cbfd83162bc85d73d69c765e7f44c478e6d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_Single

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ToStringWithCultureInfo - Duration of single invocation 132.64 ns 142.91 ns 1.08 0.01 False
ToString - Duration of single invocation 138.03 ns 147.99 ns 1.07 0.01 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Single*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Single.ToStringWithCultureInfo(value: 12345, culture: zh)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 142.90628622044207 > 140.2034244121448.
IsChangePoint: Marked as a change because one of 1/13/2023 3:30:06 PM, 2/17/2023 12:28:21 PM, 4/12/2023 8:39:35 PM, 4/20/2023 3:04:43 PM falls between 4/11/2023 11:54:05 PM and 4/20/2023 3:04:43 PM.
IsRegressionStdDev: Marked as regression because -23.24967641005557 (T) = (0 -143.45214354972157) / Math.Sqrt((0.499228183892843 / (13)) + (1.6870778247737683 / (11))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (11) - 2, .025) and -0.07639693163846963 = (133.27067305121506 - 143.45214354972157) / 133.27067305121506 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Tests.Perf_Single.ToString(value: 12345)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 147.99249155749968 > 145.38291881826893.
IsChangePoint: Marked as a change because one of 1/14/2023 12:48:25 AM, 4/12/2023 8:39:35 PM, 4/20/2023 3:04:43 PM falls between 4/11/2023 11:54:05 PM and 4/20/2023 3:04:43 PM.
IsRegressionStdDev: Marked as regression because -28.582910652915448 (T) = (0 -147.6408569304534) / Math.Sqrt((0.8187846855612998 / (13)) + (0.5088177228964171 / (10))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (10) - 2, .025) and -0.06989335876475461 = (137.99586259785008 - 147.6408569304534) / 137.99586259785008 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 0b081118dedcbc01d445f3f56222fde9d86f3f44
Compare e3f55cbfd83162bc85d73d69c765e7f44c478e6d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.Tests.Perf_BitArray

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
BitArrayIntArrayCtor - Duration of single invocation 180.55 ns 195.39 ns 1.08 0.09 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Tests.Perf_BitArray*'

Payloads

Baseline
Compare

Histogram

System.Collections.Tests.Perf_BitArray.BitArrayIntArrayCtor(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 195.38671006793535 > 190.6095051441621.
IsChangePoint: Marked as a change because one of 1/5/2023 10:03:03 PM, 1/21/2023 9:40:41 PM, 1/27/2023 7:12:54 PM, 2/6/2023 10:25:44 PM, 4/6/2023 11:23:56 PM, 4/12/2023 8:39:35 PM, 4/20/2023 3:04:43 PM falls between 4/11/2023 11:54:05 PM and 4/20/2023 3:04:43 PM.
IsRegressionStdDev: Marked as regression because -9.693736377414425 (T) = (0 -193.941444698369) / Math.Sqrt((14.675296103271831 / (13)) + (6.765563632133585 / (10))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (10) - 2, .025) and -0.07199513567397779 = (180.91634770006246 - 193.941444698369) / 180.91634770006246 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 0b081118dedcbc01d445f3f56222fde9d86f3f44
Compare e3f55cbfd83162bc85d73d69c765e7f44c478e6d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Buffers.Tests.RentReturnArrayPoolTests<Object>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SingleParallel - Duration of single invocation 9.40 μs 11.59 μs 1.23 0.43 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Buffers.Tests.RentReturnArrayPoolTests&lt;Object&gt;*'

Payloads

Baseline
Compare

Histogram

System.Buffers.Tests.RentReturnArrayPoolTests<Object>.SingleParallel(RentalSize: 4096, ManipulateArray: True, Async: True, UseSharedPool: True)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 11.588397337333332 > 9.9604774185.
IsChangePoint: Marked as a change because one of 2/14/2023 11:21:41 AM, 4/12/2023 8:39:35 PM, 4/20/2023 3:04:43 PM falls between 4/11/2023 11:54:05 PM and 4/20/2023 3:04:43 PM.
IsRegressionStdDev: Marked as regression because -4.365960155099001 (T) = (0 -11593.555546553496) / Math.Sqrt((1178285.5904132852 / (13)) + (248083.30980950376 / (10))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (10) - 2, .025) and -0.14672776391363124 = (10110.120214571429 - 11593.555546553496) / 10110.120214571429 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Apr 20, 2023 •
edited
Loading

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 0b081118dedcbc01d445f3f56222fde9d86f3f44
Compare e3f55cbfd83162bc85d73d69c765e7f44c478e6d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Json.Serialization.Tests.WriteJson<LoginViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SerializeToWriter - Duration of single invocation 286.97 ns 312.65 ns 1.09 0.03 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;LoginViewModel&gt;*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.WriteJson<LoginViewModel>.SerializeToWriter(Mode: Reflection)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 312.65253610351925 > 300.51733825310276.
IsChangePoint: Marked as a change because one of 1/20/2023 9:27:13 PM, 3/20/2023 7:45:43 PM, 4/12/2023 8:39:35 PM, 4/20/2023 3:04:43 PM falls between 4/11/2023 11:54:05 PM and 4/20/2023 3:04:43 PM.
IsRegressionStdDev: Marked as regression because -8.74058935699682 (T) = (0 -307.02216074612636) / Math.Sqrt((8.193681804342564 / (13)) + (57.65854845608221 / (10))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (10) - 2, .025) and -0.07758570859449698 = (284.91669692481133 - 307.02216074612636) / 284.91669692481133 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 0b081118dedcbc01d445f3f56222fde9d86f3f44
Compare e3f55cbfd83162bc85d73d69c765e7f44c478e6d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Memory.Span<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SequenceCompareTo - Duration of single invocation 5.96 ns 7.55 ns 1.27 0.08 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.Span&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Memory.Span<Int32>.SequenceCompareTo(Size: 4)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 7.548472357238174 > 6.254986037217673.
IsChangePoint: Marked as a change because one of 2/14/2023 11:21:41 AM, 3/3/2023 12:05:09 PM, 4/3/2023 11:26:06 PM, 4/6/2023 11:23:56 PM, 4/12/2023 8:39:35 PM, 4/20/2023 3:04:43 PM falls between 4/11/2023 11:54:05 PM and 4/20/2023 3:04:43 PM.
IsRegressionStdDev: Marked as regression because -31.494529629712773 (T) = (0 -7.528865620698522) / Math.Sqrt((0.009432752627026959 / (13)) + (0.017016498578290743 / (9))) is less than -2.0859634472534285 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (9) - 2, .025) and -0.2722144732486675 = (5.917921686170699 - 7.528865620698522) / 5.917921686170699 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 0b081118dedcbc01d445f3f56222fde9d86f3f44
Compare e3f55cbfd83162bc85d73d69c765e7f44c478e6d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Buffers.Text.Tests.Utf8FormatterTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
FormatterDecimal - Duration of single invocation 49.63 ns 70.77 ns 1.43 0.00 True
FormatterUInt32 - Duration of single invocation 2.73 ns 9.08 ns 3.32 0.75 True
FormatterInt32 - Duration of single invocation 3.09 ns 9.03 ns 2.93 0.51 False
FormatterUInt64 - Duration of single invocation 2.58 ns 9.22 ns 3.57 0.78 True

graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Buffers.Text.Tests.Utf8FormatterTests*'

Payloads

Baseline
Compare

Histogram

System.Buffers.Text.Tests.Utf8FormatterTests.FormatterDecimal(value: 123456.789)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 70.76733318661314 > 52.12096342340672.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/20/2023 3:04:43 PM falls between 4/11/2023 11:54:05 PM and 4/20/2023 3:04:43 PM.
IsRegressionStdDev: Marked as regression because -112.39100698545488 (T) = (0 -71.1270179589024) / Math.Sqrt((0.05547086874271792 / (13)) + (0.353887319330399 / (11))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (11) - 2, .025) and -0.43191007568441125 = (49.67282454864057 - 71.1270179589024) / 49.67282454864057 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Buffers.Text.Tests.Utf8FormatterTests.FormatterUInt32(value: 0)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.082373809322219 > 2.4503215097746027.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/20/2023 3:04:43 PM falls between 4/11/2023 11:54:05 PM and 4/20/2023 3:04:43 PM.
IsRegressionStdDev: Marked as regression because -64.47192468664656 (T) = (0 -9.086655711823271) / Math.Sqrt((0.1053989097468962 / (13)) + (0.027888234547839667 / (11))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (11) - 2, .025) and -2.7310035843050713 = (2.435445452276544 - 9.086655711823271) / 2.435445452276544 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt32(value: 4)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.029034535362516 > 3.1681018288817593.
IsChangePoint: Marked as a change because one of 1/21/2023 1:51:00 AM, 4/9/2023 1:38:49 AM, 4/12/2023 8:39:35 PM, 4/20/2023 3:04:43 PM falls between 4/11/2023 11:54:05 PM and 4/20/2023 3:04:43 PM.
IsRegressionStdDev: Marked as regression because -62.28571221660527 (T) = (0 -9.014487010954113) / Math.Sqrt((0.08568970937185565 / (13)) + (0.03598633166159792 / (11))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (11) - 2, .025) and -2.1867630139569867 = (2.828728390367777 - 9.014487010954113) / 2.828728390367777 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Buffers.Text.Tests.Utf8FormatterTests.FormatterUInt64(value: 0)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.219639962265049 > 2.6922796264284985.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/20/2023 3:04:43 PM falls between 4/11/2023 11:54:05 PM and 4/20/2023 3:04:43 PM.
IsRegressionStdDev: Marked as regression because -69.3801217545895 (T) = (0 -9.531514495449857) / Math.Sqrt((0.07695639190943443 / (13)) + (0.0474390136503931 / (11))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (11) - 2, .025) and -2.792340514617772 = (2.513359351226543 - 9.531514495449857) / 2.513359351226543 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 0b081118dedcbc01d445f3f56222fde9d86f3f44
Compare e3f55cbfd83162bc85d73d69c765e7f44c478e6d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_Double

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ToStringWithCultureInfo - Duration of single invocation 132.41 ns 140.93 ns 1.06 0.00 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Double*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Double.ToStringWithCultureInfo(value: 12345, culture: zh)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 140.93241925856768 > 138.45605666983352.
IsChangePoint: Marked as a change because one of 1/19/2023 4:24:48 PM, 2/17/2023 12:28:21 PM, 4/12/2023 8:39:35 PM, 4/20/2023 3:04:43 PM falls between 4/11/2023 11:54:05 PM and 4/20/2023 3:04:43 PM.
IsRegressionStdDev: Marked as regression because -24.912350274747514 (T) = (0 -141.21336131975383) / Math.Sqrt((0.905447532205026 / (13)) + (1.0411042747841968 / (11))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (13) + (11) - 2, .025) and -0.07701469057427736 = (131.11553867891732 - 141.21336131975383) / 131.11553867891732 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@kunalspathak
Copy link
Member

dotnet/runtime#84587

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@EgorBo EgorBo

Labels
ampere arch-arm64 branch-refs/heads/main kind-micro os-linux perf-regression runtime-coreclr untriaged
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@EgorBo @kunalspathak