Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added Improvements to the Features and Messaging to the GC Infra re: ASP.NET Benchmarks #3293

Merged
merged 27 commits into from
Oct 14, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
ac638f5
Added significant improvements to the ASP.NET Benchmarks
mrsharm Aug 23, 2023
de866ea
Fixed up header for the analysis
mrsharm Aug 23, 2023
049184a
Fix for the downloadOutputOutput field
mrsharm Aug 23, 2023
f80211b
Cosmetic improvement and an example of the new params
mrsharm Aug 23, 2023
6bac308
Improved messaging from the analysis from
mrsharm Aug 23, 2023
53ea22c
The create suite command now should reference the entire folder rathe…
mrsharm Aug 23, 2023
54a8b87
Fixed linting issues
mrsharm Aug 23, 2023
73dbf9f
One more Lint hydra fix
mrsharm Aug 23, 2023
aa9ae69
Fixed internal references in the markdown output
mrsharm Aug 23, 2023
8676f7a
Added example of how to use the new config
mrsharm Aug 28, 2023
6dee50c
Addressed feedback and added contextually named log files if the vari…
mrsharm Aug 30, 2023
509b410
Added the name of the file and the log file extension
mrsharm Sep 1, 2023
35b4351
Added the analysis notebook
mrsharm Sep 1, 2023
d4f29d8
Adjusted the csv files to be without spaces and modified the Linux an…
mrsharm Sep 5, 2023
fbf44c4
Updated README to reflect changes
mrsharm Sep 5, 2023
17aec7d
Updated documentation
mrsharm Sep 5, 2023
9816c4d
Fixed md liniting issues
mrsharm Sep 5, 2023
8e960ed
Fixed up the notebook to work for both windows and linux
mrsharm Sep 5, 2023
9d573d2
Continued work on improving the usability of the analysis
mrsharm Sep 9, 2023
5e9b3a2
Merge remote-tracking branch 'origin' into aspnet_improvements
mrsharm Sep 12, 2023
809b534
Added fix for certain cases where the log wasn't being downloaded
mrsharm Sep 12, 2023
8bf7c48
implemented the volatility and difference comparisons
mrsharm Sep 13, 2023
a0047ba
Incorporated feedback, detection if the process timed out and noteboo…
mrsharm Sep 15, 2023
f57042a
Cleared output
mrsharm Sep 15, 2023
7a95f42
Updated plot for vol
mrsharm Sep 15, 2023
ba8c7e0
Added benchmark filtering and fix path in the notebook
mrsharm Sep 15, 2023
82ef8a4
Ordering in terms of filters and matches on ends + regexes
mrsharm Sep 16, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view

This file was deleted.

Original file line number Diff line number Diff line change
Expand Up @@ -41,23 +41,14 @@ PlaintextPlatformInline_Windows, --config https://raw.githubusercontent.com/aspn
JsonPlatform_Windows, --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/platform.benchmarks.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/azure.profile.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/ci.profile.yml --scenario json --profile intel-win-app --profile intel-load2-load --application.collectDependencies true --application.options.collectCounters true
PlaintextPlatform_Windows, --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/platform.benchmarks.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/azure.profile.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/ci.profile.yml --scenario plaintext --profile intel-win-app --profile intel-load2-load --application.framework net8.0 --application.collectDependencies true --application.options.collectCounters true
Stage1_Windows, --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/goldilocks.benchmarks.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/ci.profile.yml --scenario basicminimalapivanilla --profile intel-win-app --profile intel-lin-load --profile amd-lin2-db --application.framework net8.0 --application.options.collectCounters true
Stage1Aot_Windows, --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/goldilocks.benchmarks.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/ci.profile.yml --scenario basicminimalapipublishaot --profile intel-win-app --profile intel-lin-load --profile amd-lin2-db --application.packageReferences "Microsoft.Dotnet.ILCompiler=$(MicrosoftNETCoreAppPackageVersion)" --application.framework net8.0 --application.options.collectCounters true
Stage1GrpcAotServerGC_Windows, --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/goldilocks.benchmarks.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/ci.profile.yml --scenario basicgrpcpublishaot --profile intel-win-app --profile intel-lin-load --application.packageReferences Microsoft.Dotnet.ILCompiler=$(MicrosoftNETCoreAppPackageVersion) --application.environmentVariables DOTNET_gcServer=1 --application.options.collectCounters true
Stage1GrpcAot_Windows, --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/goldilocks.benchmarks.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/ci.profile.yml --scenario basicgrpcpublishaot --profile intel-win-app --profile intel-lin-load --application.packageReferences Microsoft.Dotnet.ILCompiler=$(MicrosoftNETCoreAppPackageVersion) --application.options.collectCounters true
Stage1GrpcTrimR2RSingleFile_Windows, --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/goldilocks.benchmarks.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/ci.profile.yml --scenario basicgrpcpublishtrimr2rsinglefile --profile intel-win-app --profile intel-lin-load --application.options.collectCounters true
Stage1GrpcServerGC_Windows, --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/goldilocks.benchmarks.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/ci.profile.yml --scenario basicgrpcvanilla --profile intel-win-app --profile intel-lin-load --application.options.collectCounters true
Stage1Grpc_Windows, --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/goldilocks.benchmarks.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/ci.profile.yml --scenario basicgrpcvanilla --profile intel-win-app --profile intel-lin-load --application.options.collectCounters true
Stage2AotServerGC_Windows, --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/goldilocks.benchmarks.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/ci.profile.yml --scenario todosapipublishaot --profile intel-win-app --profile intel-lin-load --profile amd-lin2-db --application.packageReferences Microsoft.Dotnet.ILCompiler=$(MicrosoftNETCoreAppPackageVersion) --application.buildArguments /p:ServerGarbageCollection=true --application.options.collectCounters true
Stage2Aot_Windows, --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/goldilocks.benchmarks.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/ci.profile.yml --scenario todosapipublishaot --profile intel-win-app --profile intel-lin-load --profile amd-lin2-db --application.packageReferences Microsoft.Dotnet.ILCompiler=$(MicrosoftNETCoreAppPackageVersion) --application.options.collectCounters true
Stage2TrimR2RSingleFile_Windows, --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/goldilocks.benchmarks.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/ci.profile.yml --scenario todosapipublishtrimr2rsinglefile --profile intel-win-app --profile intel-lin-load --profile amd-lin2-db --application.options.collectCounters true
Stage2ServerGC_Windows, --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/goldilocks.benchmarks.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/ci.profile.yml --scenario todosapivanilla --profile intel-win-app --profile intel-lin-load --profile amd-lin2-db --application.buildArguments /p:ServerGarbageCollection=true --application.options.collectCounters true
Stage2_Windows, --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/goldilocks.benchmarks.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/ci.profile.yml --scenario todosapivanilla --profile intel-win-app --profile intel-lin-load --profile amd-lin2-db --application.options.collectCounters true
Stage1AotSpeedOpt_Windows, --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/goldilocks.benchmarks.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/ci.profile.yml --scenario basicminimalapipublishaot --profile intel-win-app --profile intel-lin-load --profile amd-lin2-db --application.packageReferences Microsoft.Dotnet.ILCompiler=$(MicrosoftNETCoreAppPackageVersion) --application.buildArguments /p:OptimizationPreference=Speed --application.options.collectCounters true
Stage1AotServerGC_Windows, --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/goldilocks.benchmarks.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/ci.profile.yml --scenario basicminimalapipublishaot --profile intel-win-app --profile intel-lin-load --application.packageReferences Microsoft.Dotnet.ILCompiler=$(MicrosoftNETCoreAppPackageVersion) --application.environmentVariables DOTNET_gcServer=1 --application.options.collectCounters true
Stage1TrimR2RSingleFile_Windows, --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/goldilocks.benchmarks.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/ci.profile.yml --scenario basicminimalapipublishtrimr2rsinglefile --profile intel-win-app --profile intel-lin-load --application.options.collectCounters true
Stage1ServerGC_Windows, --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/goldilocks.benchmarks.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/ci.profile.yml --scenario basicminimalapivanilla --profile intel-win-app --profile intel-lin-load --application.environmentVariables DOTNET_gcServer=1 --application.options.collectCounters true
Stage1GrpcPgo_Windows, --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/goldilocks.benchmarks.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/ci.profile.yml --scenario basicgrpcvanilla --profile intel-win-app --profile intel-lin-load --application.environmentVariables DOTNET_TieredPGO=1 --application.options.collectCounters true
Stage2Pgo_Windows, --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/goldilocks.benchmarks.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/ci.profile.yml --scenario todosapivanilla --profile intel-win-app --profile intel-lin-load --profile amd-lin2-db --application.environmentVariables DOTNET_TieredPGO=1 --application.options.collectCounters true
Stage1Pgo_Windows, --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/goldilocks.benchmarks.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/ci.profile.yml --scenario basicminimalapivanilla --profile intel-win-app --profile intel-lin-load --application.environmentVariables DOTNET_TieredPGO=1 --application.options.collectCounters true
Stage1GrpcAotSpeedOpt_Windows, --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/goldilocks.benchmarks.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/ci.profile.yml --scenario basicgrpcpublishaot --profile intel-win-app --profile intel-lin-load --profile amd-lin2-db --application.packageReferences Microsoft.Dotnet.ILCompiler=$(MicrosoftNETCoreAppPackageVersion) --application.buildArguments /p:OptimizationPreference=Speed --application.options.collectCounters true
Stage2AotSpeedOpt_Windows, --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/goldilocks.benchmarks.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/ci.profile.yml --scenario todosapipublishaot --profile intel-win-app --profile intel-lin-load --profile amd-lin2-db --application.packageReferences Microsoft.Dotnet.ILCompiler=$(MicrosoftNETCoreAppPackageVersion) --application.buildArguments /p:OptimizationPreference=Speed --application.options.collectCounters true
Stage1Pgo_Windows, --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/goldilocks.benchmarks.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/ci.profile.yml --scenario basicminimalapivanilla --profile intel-win-app --profile intel-lin-load --application.environmentVariables DOTNET_TieredPGO=1 --application.options.collectCounters true
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
Legend,Base CommandLine
Stage1Aot_Windows, --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/goldilocks.benchmarks.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/ci.profile.yml --scenario basicminimalapipublishaot --profile intel-win-app --profile intel-lin-load --profile amd-lin2-db --application.packageReferences "Microsoft.Dotnet.ILCompiler=$(MicrosoftNETCoreAppPackageVersion)" --application.framework net8.0 --application.options.collectCounters true
Stage1GrpcAot_Windows, --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/goldilocks.benchmarks.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/ci.profile.yml --scenario basicgrpcpublishaot --profile intel-win-app --profile intel-lin-load --application.packageReferences Microsoft.Dotnet.ILCompiler=$(MicrosoftNETCoreAppPackageVersion) --application.options.collectCounters true
Stage1AotSpeedOpt_Windows, --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/goldilocks.benchmarks.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/ci.profile.yml --scenario basicminimalapipublishaot --profile intel-win-app --profile intel-lin-load --profile amd-lin2-db --application.packageReferences Microsoft.Dotnet.ILCompiler=$(MicrosoftNETCoreAppPackageVersion) --application.buildArguments /p:OptimizationPreference=Speed --application.options.collectCounters true
Stage1GrpcAotSpeedOpt_Windows, --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/goldilocks.benchmarks.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/ci.profile.yml --scenario basicgrpcpublishaot --profile intel-win-app --profile intel-lin-load --profile amd-lin2-db --application.packageReferences Microsoft.Dotnet.ILCompiler=$(MicrosoftNETCoreAppPackageVersion) --application.buildArguments /p:OptimizationPreference=Speed --application.options.collectCounters true
Stage2Aot_Windows, --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/goldilocks.benchmarks.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/ci.profile.yml --scenario todosapipublishaot --profile intel-win-app --profile intel-lin-load --profile amd-lin2-db --application.packageReferences Microsoft.Dotnet.ILCompiler=$(MicrosoftNETCoreAppPackageVersion) --application.options.collectCounters true
Stage2AotSpeedOpt_Windows, --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/goldilocks.benchmarks.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/ci.profile.yml --scenario todosapipublishaot --profile intel-win-app --profile intel-lin-load --profile amd-lin2-db --application.packageReferences Microsoft.Dotnet.ILCompiler=$(MicrosoftNETCoreAppPackageVersion) --application.buildArguments /p:OptimizationPreference=Speed --application.options.collectCounters true
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
Legend,Base CommandLine
FortunesEf_Linux, --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/database.benchmarks.yml --scenario fortunes_ef --application.framework net8.0 --application.options.collectCounters true --profile aspnet-citrine-lin --property os=linux --property arch=x64
JsonMin_Linux, --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/json.benchmarks.yml --scenario mapaction --application.framework net8.0 --application.options.collectCounters true --profile aspnet-citrine-lin --property os=linux --property arch=x64
Stage1Grpc_Linux, crank --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/goldilocks.benchmarks.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/ci.profile.yml --scenario basicgrpcvanilla --profile intel-load2-app --profile amd-lin2-load --profile amd-lin2-db --application.options.collectCounters true
Original file line number Diff line number Diff line change
@@ -1,2 +1,3 @@
Legend,Base CommandLine
Stage1Grpc_Linux, --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/goldilocks.benchmarks.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/ci.profile.yml --scenario basicgrpcvanilla --profile intel-load2-app --profile amd-lin2-load --profile amd-lin2-db --application.options.collectCounters true
JsonMvc_Windows, --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/json.benchmarks.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/azure.profile.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/ci.profile.yml --scenario mvc --profile intel-win-app --profile intel-load2-load --application.collectDependencies true --application.options.collectCounters true
FortunesEf_Windows, --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/database.benchmarks.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/azure.profile.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/ci.profile.yml --scenario fortunes_ef --profile intel-win-app --profile intel-load-load --profile intel-db-db --application.options.collectCounters true --application.collectDependencies true
Original file line number Diff line number Diff line change
Expand Up @@ -3,15 +3,23 @@ runs:
corerun: C:\CoreRuns\Empty\
environment_variables:
COMPlus_GCServer: 1
framework_version: net6.0
mrsharm marked this conversation as resolved.
Show resolved Hide resolved
run:
corerun: C:\CoreRuns\Empty\
environment_variables:
COMPlus_GCServer: 1
environment:
environment_variables: {}
default_max_seconds: 300
framework_version: net8.0
benchmark_settings:
benchmark_file: C:\InfraRuns\RunNew_All\Suites\ASPNETBenchmarks\ASPNetBenchmarks.csv
# To take a dump: --application.options.dumpType full --application.options.dumpOutput <filename>
# To fetch the build artifacts: --application.options.fetch true
additional_arguments: --chart --chart-type hex
# Can optionally filter for specific benchmarks with a list of regexes.
# benchmark_filters:
# - Platform*
output:
path: C:\InfraRuns\RunNew_All\ASPNetBenchmarks
columns:
Expand Down
Loading
Loading