diff --git a/docs/benchmarking_docker.md b/docs/benchmarking_docker.md index a0e450a2f8..c59a42d93f 100644 --- a/docs/benchmarking_docker.md +++ b/docs/benchmarking_docker.md @@ -22,12 +22,13 @@ variables: jobs: server: - source: - repository: https://github.com/TechEmpower/FrameworkBenchmarks - branchOrCommit: master - dockerFile: frameworks/Java/netty/netty.dockerfile - dockerImageName: netty - dockerContextDirectory: frameworks/Java/netty + sources: + techempower: + repository: https://github.com/TechEmpower/FrameworkBenchmarks + branchOrCommit: master + dockerFile: techempower/frameworks/Java/netty/netty.dockerfile + dockerImageName: netty + dockerContextDirectory: techempower/frameworks/Java/netty port: 8080 scenarios: @@ -70,25 +71,26 @@ By default it will assume the agent is listening on `http://localhost:5010`. In ## Understanding the Docker configuration -The `jobs` section in the configuration file defines a job named `server`. It's `source` property contains the information necessary to setup the Docker container. +The `jobs` section in the configuration file defines a job named `server`. It's `sources` property contains the information about where to download any sources necessary for the job. In this case there is a single source named `techempower` and so the source will be copied into a directory called `techempower` in the job's working directory. The `dockerFile`, `dockerImageName`, `dockerContextDirectory` are then used to define what docker command is run. In this case it is a **Netty** application taken from the [TechEmpower](https://github.com/TechEmpower/FrameworkBenchmarks) repository which contains several types of benchmarks for any language and framework. Netty is a Java web application framework. ```yml server: - source: - repository: https://github.com/TechEmpower/FrameworkBenchmarks - branchOrCommit: master - dockerFile: frameworks/Java/netty/netty.dockerfile - dockerImageName: netty - dockerContextDirectory: frameworks/Java/netty + sources: + techempower: + repository: https://github.com/TechEmpower/FrameworkBenchmarks + branchOrCommit: master + dockerFile: techempower/frameworks/Java/netty/netty.dockerfile + dockerImageName: netty + dockerContextDirectory: techempower/frameworks/Java/netty port: 8080 ``` -The `dockerFile` property is relative to the root of the repository, and point to the Docker file to build. +The `dockerFile` property is relative to the job's working directory, and points to the Docker file to build. Note that the path starts with `techempower` because each source is copied to a directory with the name of the source. The `dockerImageName` is used on the agent to name the container. By setting it Docker will be able to reuse previous cached containers and start benchmarks faster. -The `dockerContextDirectory` is a path representing the working directory inside the Docker file script. Any path using `./` in the Docker file will be this value. It's relative to the root of the source (the clone of the repository). +The `dockerContextDirectory` is a path representing the working directory inside the Docker file script. Any path using `./` in the Docker file will be this value. It's relative to the job's working directory. The property `port` is used to let the Crank agent detect when the application is ready to accept requests. When specified, the agent will try to ping the expected endpoint until it successfully answers. Another option is to define the property `readyStateText` with some value that is written on the standard output once the application has started. In the case of a Netty application, the service emits `Httpd started. Listening on: 0.0.0.0/0.0.0.0:8080` when the service is ready. Similarly, an ASP.NET application outputs `Application started. Press Ctrl+C to shut down.` diff --git a/docs/getting_started.md b/docs/getting_started.md index e548153b3f..41ef25ad2c 100644 --- a/docs/getting_started.md +++ b/docs/getting_started.md @@ -37,10 +37,11 @@ imports: jobs: server: - source: - repository: https://github.com/dotnet/crank - branchOrCommit: main - project: samples/hello/hello.csproj + sources: + crank: + repository: https://github.com/dotnet/crank + branchOrCommit: main + project: crank/samples/hello/hello.csproj scenarios: hello: diff --git a/docs/local_application.md b/docs/local_application.md index 565fdc00d3..3c648ca422 100644 --- a/docs/local_application.md +++ b/docs/local_application.md @@ -10,32 +10,36 @@ This tutorial explains how to send local source files to an agent instead of clo ## Define the scenario -In the [Getting Started](getting_started.md) tutorial, the **hello** application is benchmarked. It's `source` section is pointing to the Crank repository to inform the Crank Agent to clone it and build it. +In the [Getting Started](getting_started.md) tutorial, the **hello** application is benchmarked. It's `sources` section has a source named `crank` which is pointing to the Crank repository to inform the Crank Agent to clone it. By default, a source will be cloned to a directory with the same name as the source. + +The `project` property defines a path to a .NET project file that should be built and run as the work carried out by this job. The project's path starts with `crank` because that is the folder that the source is cloned to, and `samples/hello/hello.csproj` is where the project file is located inside that source. ```yml server: - source: - repository: https://github.com/dotnet/crank - branchOrCommit: main - project: samples/hello/hello.csproj + sources: + crank: + repository: https://github.com/dotnet/crank + branchOrCommit: main + project: crank/samples/hello/hello.csproj readyStateText: Application started. ``` In cases where you want to iterate quickly on an application, it's easier to do changes locally and send the source file to the agent instead of having it clone changes what you would have to push. -The `source` property of a job has a `localFolder` property that can be set to a local folder. +A source inside the `sources` property can specify a `localFolder` property that can be set to a local folder. The file `/crank/samples/local/local.benchmarks.yml` demonstrates how to use this property instead. ```yml server: - source: - localFolder: ../../samples/hello - project: hello.csproj + sources: + hello-sample: + localFolder: ../../samples/hello + project: hello-sample/hello.csproj readyStateText: Application started. ``` -The path is relative to the configuration file that contains it. +The `localFolder` path is relative to the .yml configuration file that contains it. Run the following command line to execute this job. diff --git a/docs/microbenchmarks.md b/docs/microbenchmarks.md index 7a117bb571..3f468af315 100644 --- a/docs/microbenchmarks.md +++ b/docs/microbenchmarks.md @@ -50,9 +50,10 @@ The following example is available at https://github.com/dotnet/crank/blob/main/ ```yml benchmarks: - source: - localFolder: . - project: micro.csproj + sources: + micro: + localFolder: . + project: micro/micro.csproj variables: filterArg: "*" jobArg: short diff --git a/docs/precommands.md b/docs/precommands.md index 14e6b2f98b..cf0552b6b3 100644 --- a/docs/precommands.md +++ b/docs/precommands.md @@ -25,10 +25,11 @@ commands: jobs: server: - source: - localFolder: ../../artifacts/bin/hello/Release/net8.0 + sources: + hello: + localFolder: ../../artifacts/bin/hello/Release/net8.0 executable: dotnet - arguments: hello.dll + arguments: hello/hello.dll noBuild: true beforeJob: - buildHello diff --git a/docs/reference/architecture.md b/docs/reference/architecture.md index 83add4a23d..f7ff44f80e 100644 --- a/docs/reference/architecture.md +++ b/docs/reference/architecture.md @@ -16,10 +16,11 @@ imports: jobs: server: - source: - repository: https://github.com/dotnet/crank - branchOrCommit: main - project: samples/hello/hello.csproj + sources: + crank: + repository: https://github.com/dotnet/crank + branchOrCommit: main + project: crank/samples/hello/hello.csproj scenarios: hello: diff --git a/samples/hello/hello.benchmarks.yml b/samples/hello/hello.benchmarks.yml index 7c02f391a8..0fd284aae0 100644 --- a/samples/hello/hello.benchmarks.yml +++ b/samples/hello/hello.benchmarks.yml @@ -3,10 +3,11 @@ imports: jobs: server: - source: - repository: https://github.com/dotnet/crank - branchOrCommit: main - project: samples/hello/hello.csproj + sources: + crank: + repository: https://github.com/dotnet/crank + branchOrCommit: main + project: crank/samples/hello/hello.csproj readyStateText: Application started. scenarios: diff --git a/samples/local/local.benchmarks.yml b/samples/local/local.benchmarks.yml index 5d5e179533..a0d3f0e824 100644 --- a/samples/local/local.benchmarks.yml +++ b/samples/local/local.benchmarks.yml @@ -3,9 +3,10 @@ imports: jobs: server: - source: - localFolder: ../../samples/hello - project: hello.csproj + sources: + hello-sample: + localFolder: ../../samples/hello + project: hello-sample/hello.csproj readyStateText: Application started. scenarios: diff --git a/samples/micro/dotnet.benchmarks.yml b/samples/micro/dotnet.benchmarks.yml index fb2f8047bb..a9d9605415 100644 --- a/samples/micro/dotnet.benchmarks.yml +++ b/samples/micro/dotnet.benchmarks.yml @@ -1,9 +1,10 @@ jobs: dotnet: - source: - repository: https://github.com/dotnet/performance - branchOrCommit: main - project: src/benchmarks/micro/MicroBenchmarks.csproj + sources: + performance: + repository: https://github.com/dotnet/performance + branchOrCommit: main + project: performance/src/benchmarks/micro/MicroBenchmarks.csproj variables: filterArg: "*" jobArg: short diff --git a/samples/micro/micro.benchmarks.yml b/samples/micro/micro.benchmarks.yml index adec64334a..57066a996a 100644 --- a/samples/micro/micro.benchmarks.yml +++ b/samples/micro/micro.benchmarks.yml @@ -1,8 +1,9 @@ jobs: benchmarks: - source: - localFolder: . - project: micro.csproj + sources: + micro: + localFolder: . + project: micro/micro.csproj variables: filterArg: "*" jobArg: short diff --git a/samples/netty/netty.benchmarks.yml b/samples/netty/netty.benchmarks.yml index 559fe4f75c..b9b6d6a462 100644 --- a/samples/netty/netty.benchmarks.yml +++ b/samples/netty/netty.benchmarks.yml @@ -6,12 +6,13 @@ variables: jobs: server: - source: - repository: https://github.com/TechEmpower/FrameworkBenchmarks - branchOrCommit: master - dockerFile: frameworks/Java/netty/netty.dockerfile - dockerImageName: netty - dockerContextDirectory: frameworks/Java/netty + sources: + techempower: + repository: https://github.com/TechEmpower/FrameworkBenchmarks + branchOrCommit: master + dockerFile: techempower/frameworks/Java/netty/netty.dockerfile + dockerImageName: netty + dockerContextDirectory: techempower/frameworks/Java/netty port: 8080 scenarios: diff --git a/samples/precommand/precommand.benchmarks.yml b/samples/precommand/precommand.benchmarks.yml index c5e7a1dd1d..d8b39229a1 100644 --- a/samples/precommand/precommand.benchmarks.yml +++ b/samples/precommand/precommand.benchmarks.yml @@ -13,10 +13,11 @@ commands: jobs: server: - source: - localFolder: ../../artifacts/bin/hello/Release/net8.0 + sources: + hello: + localFolder: ../../artifacts/bin/hello/Release/net8.0 executable: dotnet - arguments: hello.dll + arguments: hello/hello.dll noBuild: true beforeJob: - buildHello diff --git a/src/Microsoft.Crank.Controller/Documentation.cs b/src/Microsoft.Crank.Controller/Documentation.cs index 0b597b8306..f4b8db11b1 100644 --- a/src/Microsoft.Crank.Controller/Documentation.cs +++ b/src/Microsoft.Crank.Controller/Documentation.cs @@ -11,14 +11,14 @@ These options are specific to a job instance. Replace [JOB] by the name of the s ## Sources location - --[JOB].source.repository The git repository containing the source code to build, e.g., https://github.com/aspnet/aspnetcore - --[JOB].source.branchOrCommit A branch name or commit hash, e.g., my/branch, my/branch#commit-hash - --[JOB].source.initSubmodules Whether to init submodules when a git repository is used, e.g., true - --[JOB].source.localFolder The local path containing the source code to upload to the server. e.g., /code/mybenchmarks + --[JOB].sources.[SOURCE].repository The git repository containing the source code to build, e.g., https://github.com/aspnet/aspnetcore + --[JOB].sources.[SOURCE].branchOrCommit A branch name or commit hash, e.g., my/branch, my/branch#commit-hash + --[JOB].sources.[SOURCE].initSubmodules Whether to init submodules when a git repository is used, e.g., true + --[JOB].sources.[SOURCE].localFolder The local path containing the source code to upload to the server. e.g., /code/mybenchmarks ## .NET options - --[JOB].source.project The project file to build, relative to the source code base path, e.g., src/Benchmarks/Benchmarks.csproj + --[JOB].project The project file to build, relative to the source code base path, e.g., src/Benchmarks/Benchmarks.csproj --[JOB].sdkVersion The version of the .NET SDK to install and use. By default the latest available build is used. --[JOB].runtimeVersion The version of the .NET runtime to install and use. It is defined as MicrosoftNETCoreAppPackageVersion in the build arguments. By default the latest available build is used. Setting this value forces the app to @@ -37,13 +37,13 @@ be deployed as stand-alone. ## Docker options - --[JOB].source.dockerFile The local path to the Docker file, e.g., frameworks/Rust/actix/actix-raw.dockerfile - --[JOB].source.dockerImageName The name of the docker image to create, e.g., actix_raw - --[JOB].source.dockerContextDirectory The folder in which the Docker file is built relative to, e.g., frameworks/Rust/actix/ - --[JOB].source.dockerFetchPath The path in the Docker container that contains the base path for the --fetch option, e.g., ./output - --[JOB].source.dockerLoad The path of an image to use for 'docker load', e.g, ""./myimage.tar"" - --[JOB].source.dockerPull The image name to pull and run, e.g, ""redis"", ""mcr.microsoft.com/dotnet/aspnet:7.0"" - --[JOB].source.dockerCommand The 'docker run' command, e.g, ""./startup.sh"" + --[JOB].dockerFile The local path to the Docker file, e.g., frameworks/Rust/actix/actix-raw.dockerfile + --[JOB].dockerImageName The name of the docker image to create, e.g., actix_raw + --[JOB].dockerContextDirectory The folder in which the Docker file is built relative to, e.g., frameworks/Rust/actix/ + --[JOB].dockerFetchPath The path in the Docker container that contains the base path for the --fetch option, e.g., ./output + --[JOB].dockerLoad The path of an image to use for 'docker load', e.g, ""./myimage.tar"" + --[JOB].dockerPull The image name to pull and run, e.g, ""redis"", ""mcr.microsoft.com/dotnet/aspnet:7.0"" + --[JOB].dockerCommand The 'docker run' command, e.g, ""./startup.sh"" --[JOB].buildArguments An argument to pass to 'docker build' as a '--build-arg' value. Can be used multiple times to define multiple values. @@ -88,7 +88,7 @@ out of 1200% total available CPU time. ## Debugging --[JOB].noClean Whether to keep the work folder on the server or not. Default is false, such that each job is cleaned once it's finished. - --[JOB].options.fetch Whether the benchmark folder is downloaded. e.g., true. For Docker see '--[JOB].source.dockerFetchPath' + --[JOB].options.fetch Whether the benchmark folder is downloaded. e.g., true. For Docker see '--[JOB].dockerFetchPath' --[JOB].options.fetchOutput The name of the fetched archive. Can be a file prefix (app will add *.DATE*.zip) , or a specific name (end in *.zip) and no DATE* will be added e.g., c:\publishedapps\myApp --[JOB].options.displayOutput Whether to download and display the standard output of the benchmark. --[JOB].options.displayBuild Whether to download and display the standard output of the build step (works for .NET and Docker). diff --git a/src/Microsoft.Crank.Controller/Program.cs b/src/Microsoft.Crank.Controller/Program.cs index df3b38580b..1888a7d57d 100644 --- a/src/Microsoft.Crank.Controller/Program.cs +++ b/src/Microsoft.Crank.Controller/Program.cs @@ -2377,18 +2377,14 @@ public static async Task LoadConfigurationAsync(string configurationFil var jobObject = (JObject)job.Value; if (jobObject.ContainsKey("source")) { - var source = (JObject)jobObject["source"]; - if (source.ContainsKey("localFolder")) - { - var localFolder = source["localFolder"].ToString(); - - if (!localFolder.StartsWith("http", StringComparison.OrdinalIgnoreCase)) - { - var configurationFilename = new FileInfo(configurationFilenameOrUrl).FullName; - var resolvedFilename = new FileInfo(Path.Combine(Path.GetDirectoryName(configurationFilename), localFolder)).FullName; + PatchLocalFolderInSource(configurationFilenameOrUrl, (JObject)jobObject["source"]); + } - source["localFolder"] = resolvedFilename; - } + if (jobObject.ContainsKey("sources")) + { + foreach (JProperty source in jobObject["sources"]) + { + PatchLocalFolderInSource(configurationFilenameOrUrl, (JObject)source.Value); } } } @@ -2430,6 +2426,22 @@ public static async Task LoadConfigurationAsync(string configurationFil } } + private static void PatchLocalFolderInSource(string configurationFilenameOrUrl, JObject source) + { + if (source.ContainsKey("localFolder")) + { + var localFolder = source["localFolder"].ToString(); + + if (!localFolder.StartsWith("http", StringComparison.OrdinalIgnoreCase)) + { + var configurationFilename = new FileInfo(configurationFilenameOrUrl).FullName; + var resolvedFilename = new FileInfo(Path.Combine(Path.GetDirectoryName(configurationFilename), localFolder)).FullName; + + source["localFolder"] = resolvedFilename; + } + } + } + /// /// Merges a JObject into another one. /// diff --git a/src/Microsoft.Crank.Controller/README.md b/src/Microsoft.Crank.Controller/README.md index 05137df5d5..2358285a52 100644 --- a/src/Microsoft.Crank.Controller/README.md +++ b/src/Microsoft.Crank.Controller/README.md @@ -3,133 +3,152 @@ ## Usage ``` -Usage: crank [options] +Usage: crank [command] [options] Options: - -?|-h|--help Show help information - - These options are not specific to a job instance. - - --config Configuration file or url. Can be used multiple times. - --scenario Scenario to execute. - --profile Profiles to apply. Can be used multiple times. - --variable A global variable. Can be used multiple times. e.g. --variable connections=16 - --variable-json A global variable in JSON format. Can be used multiple times. e.g. --variable-json "customHeaders=['accept-encoding:deflat']" - --compare An optional filename to compare the results to. Can be used multiple times. - - --session A logical identifier to group related jobs. - --description A string describing the job. - --property Some custom key/value that will be added to the results, e.g. --property arch=arm --property os=linux - - --sql Connection string or environment variable name of the SQL Server Database to store results in. - --table Table or environment variable name of the SQL Database to store results in. - --es Url or environment variable name of the Elasticsearch server to store results in. - --index Index name or environment variable name of the Elasticsearch server to store results in. - --json Store the results as json in the specified file. - --csv Store or append the results as csv in the specified file. - --no-metadata Don't record metadata in the stored document. - --no-measurements Don't record measurements in the stored document. - - -m|--interval The measurements interval in seconds. Default is 1. - -i|--iterations The number of times to repeat the jobs to get average results. By default the last job only is repeated, unless '--repeat' is set. - --repeat The job to repeat using the '--duration' or '--iterations' argument. - --span The duration while the job is repeated. - --auto-flush Runs a single long-running job and flushes measurements automatically. - -x|--exclude Excludes the specified number of high and low results, e.g., 1, 1:0 (exclude the lowest), 0:3 (exclude the 3 highest) - -xo|--exclude-order The result to use to detect the high and low results, e.g., 'load:http/rps/mean' - --chart Renders a chart for multi-value results. - --chart-type [bar (default) | hex] Type of chart to render. Values are 'bar' (default) or 'hex' - --chart-scale [off (default)| auto] Scale for chart. Values are 'off' (default) or 'auto'. When scale is off, the min value starts at 0. - --script [name] Execute a named script available in the configuration files. Can be used multiple times. - --debug Saves the final configuration to a file and skips the execution of the benchmark, e.g., '-d debug.json' - --relay Connection string or environment variable name of the Azure Relay namespace used to access the Crank Agent endpoints. e.g., 'Endpoint=sb://mynamespace.servicebus.windows.net;...', 'MY_AZURE_RELAY_ENV'; - --command-line-property Saves the final crank command line in a custom 'command-line' property, excluding all unnecessary and security sensitive arguments. - - These options are specific to a job instance. Replace [JOB] by the name of the service in the scenario (usually "application"). - - --[JOB].endpoints An endpoint on which to deploy the job definition, e.g., http://asp-perf-lin:5001. Can be used multiple times. + -?|-h|--help Show help information + -c|--config Configuration file or url. + -s|--scenario Scenario to execute. + -j|--job Creates a job that is not defined in configuration. + --profile Profile name. + --script Execute a named script available in the configuration files. Can be used multiple times. + -j|--json Saves the results as json in the specified file. + --csv Saves the results as csv in the specified file. + --compare An optional filename to compare the results to. Can be used multiple times. + --variable Variable + --variable-json Typed Variable + --sql Connection string or environment variable name of the SQL Server Database to store results in. + --table Table name or environment variable name of the SQL table to store results in. + --es Elasticsearch server url to store results in. + --index Index name of the Elasticsearch server to store results in. + --relay Connection string or environment variable name of the Azure Relay namespace used to access the Crank Agent endpoints. e.g., 'Endpoint=sb://mynamespace.servicebus.windows.net;...', 'MY_AZURE_RELAY_ENV' + --session A logical identifier to group related jobs. + --description A string describing the job. + -p|--property Some custom key/value that will be added to the results, .e.g. --property arch=arm --property os=linux + --no-measurements Remove all measurements from the stored results. For instance, all samples of a measure won't be stored, only the final value. + --no-metadata Remove all metadata from the stored results. The metadata is only necessary for being to generate friendly outputs. + --auto-flush Runs a single long-running job and flushes measurements automatically. + --repeat The job to repeat using the '--span' or '--iterations' argument. + --span The duration while the job is repeated. + --chart Renders a chart for multi-value results. + --chart-type Type of chart to render. Values are 'bar' (default) or 'hex' + --chart-scale Scale for chart. Values are 'off' (default) or 'auto'. When scale is off, the min value starts at 0. + -i|--iterations The number of iterations. + -m|--interval The measurements interval in seconds. Default is 1. + -v|--verbose Verbose output + --quiet Quiet output, only the results are displayed + -x|--exclude Excludes the specified number of high and low results, e.g., 1, 1:0 (exclude the lowest), 0:3 (exclude the 3 highest) + -xo|--exclude-order The result to use to detect the high and low results, e.g., 'load:http/rps/mean' + -d|--debug Saves the final configuration to a file and skips the execution of the benchmark, e.g., '-d debug.json' + --command-line-property Saves the final crank command line in a custom 'command-line' property, excludinf all unnecessary and security sensitive arguments. + +Commands: + compare Compares result files + +Run 'crank [command] -?|-h|--help' for more information about a command. + + These options are specific to a job instance. Replace [JOB] by the name of the service in the scenario (usually "application"). + + --[JOB].endpoints Adds an endpoint on which to deploy the job definition, e.g., http://asp-perf-lin:5001. ## Sources location - --[JOB].source.repository The git repository containing the source code to build, e.g., https://github.com/aspnet/aspnetcore - --[JOB].source.branchOrCommit A branch name or commit hash, e.g., my/branch, my/branch#commit-hash - --[JOB].source.initSubmodules Whether to init submodules when a git repository is used, e.g., true - --[JOB].source.localFolder The local path containing the source code to upload to the server. e.g., /code/mybenchmarks + --[JOB].sources.[SOURCE].repository The git repository containing the source code to build, e.g., https://github.com/aspnet/aspnetcore + --[JOB].sources.[SOURCE].branchOrCommit A branch name or commit hash, e.g., my/branch, my/branch#commit-hash + --[JOB].sources.[SOURCE].initSubmodules Whether to init submodules when a git repository is used, e.g., true + --[JOB].sources.[SOURCE].localFolder The local path containing the source code to upload to the server. e.g., /code/mybenchmarks ## .NET options - --[JOB].source.project The project file to build, relative to the source code base path, e.g., src/Benchmarks/Benchmarks.csproj - --[JOB].sdkVersion The version of the .NET SDK to install and use. By default the latest available build is used. - --[JOB].runtimeVersion The version of the .NET runtime to install and use. It is defined as MicrosoftNETCoreAppPackageVersion in the build arguments. By default the latest available build is used. Setting this value forces the app to be deployed as stand-alone. - --[JOB].aspNetCoreVersion The version of the ASP.NET runtime to install and use. It is defined as MicrosoftAspNetCoreAppPackageVersion in the build arguments. By default the latest available build is used. Setting this value forces the app to be deployed as stand-alone. - --[JOB].noGlobalJson Whether to not emit any global.json file to force the .NET SDK version to use. Default is false, meaning whatever version of the .NET SDK is chosen, it will be set in a global.json file. - --[JOB].framework The framework version to use in case it can't be assumed from the .NET runtime version. e.g., net8.0 - --[JOB].buildArguments An argument to pass to msbuild. Can be used multiple times to define multiple values. - --[JOB].selfContained Whether to deploy the app as stand-alone. Default is true. - --[JOB].useMonoRuntime Use a specific mono runtime instead of the dotnet core runtime. - --[JOB].packageReferences A package reference to add to the csproj. Can be used multiple times to define multiple values. - --[JOB].patchReferences Whether to patch the TFM of project references. Default is false. + --[JOB].project The project file to build, relative to the source code base path, e.g., src/Benchmarks/Benchmarks.csproj + --[JOB].sdkVersion The version of the .NET SDK to install and use. By default the latest available build is used. + --[JOB].runtimeVersion The version of the .NET runtime to install and use. It is defined as MicrosoftNETCoreAppPackageVersion + in the build arguments. By default the latest available build is used. Setting this value forces the app to + be deployed as stand-alone. + --[JOB].aspNetCoreVersion The version of the ASP.NET runtime to install and use. It is defined as MicrosoftAspNetCoreAppPackageVersion + in the build arguments. By default the latest available build is used. Setting this value forces the app to + be deployed as stand-alone. + --[JOB].noGlobalJson Whether to not emit any global.json file to force the .NET SDK version to use. Default is false, meaning + whatever version of the .NET SDK is chosen, it will be set in a global.json file. + --[JOB].framework The framework version to use in case it can't be assumed from the .NET runtime version. e.g., net8.0 + --[JOB].buildArguments An argument to pass to msbuild. Can be used multiple times to define multiple values. + --[JOB].selfContained Whether to deploy the app as stand-alone. Default is true. + --[JOB].useMonoRuntime Use a specific mono runtime instead of the dotnet core runtime. + --[JOB].packageReferences A package reference to add to the csproj. Can be used multiple times to define multiple values. + --[JOB].patchReferences Whether to patch the TFM of project references. Default is false. ## Docker options - --[JOB].source.dockerFile The local path to the Docker file, e.g., frameworks/Rust/actix/actix-raw.dockerfile - --[JOB].source.dockerImageName The name of the docker image to create, e.g., actix_raw - --[JOB].source.dockerContextDirectory The folder in which the Docker file is built relative to, e.g., frameworks/Rust/actix/ - --[JOB].source.dockerFetchPath The path in the Docker container that contains the base path for the --fetch option, e.g., ./output - --[JOB].source.dockerLoad The path of an image to use for 'docker load', e.g, "./myimage.tar" - --[JOB].source.dockerPull The image name to pull and run, e.g, "redis", "mcr.microsoft.com/dotnet/aspnet:7.0" - --[JOB].source.dockerCommand The 'docker run' command, e.g, "./startup.sh" - --[JOB].buildArguments An argument to pass to 'docker build' as a '--build-arg' value. Can be used multiple times to define multiple values. + --[JOB].dockerFile The local path to the Docker file, e.g., frameworks/Rust/actix/actix-raw.dockerfile + --[JOB].dockerImageName The name of the docker image to create, e.g., actix_raw + --[JOB].dockerContextDirectory The folder in which the Docker file is built relative to, e.g., frameworks/Rust/actix/ + --[JOB].dockerFetchPath The path in the Docker container that contains the base path for the --fetch option, e.g., ./output + --[JOB].dockerLoad The path of an image to use for 'docker load', e.g, "./myimage.tar" + --[JOB].dockerPull The image name to pull and run, e.g, "redis", "mcr.microsoft.com/dotnet/aspnet:7.0" + --[JOB].dockerCommand The 'docker run' command, e.g, "./startup.sh" + --[JOB].buildArguments An argument to pass to 'docker build' as a '--build-arg' value. Can be used multiple times to define multiple + values. ## Diagnostics - --[JOB].dotnetTrace Whether to collect a diagnostics trace using dotnet-trace. An optional profile name or list of dotnet-trace providers can be passed. e.g., true - --[JOB].dotnetTraceProviders A comma-separated list of trace providers. By default the profile 'cpu-sampling' is used. See https://github.com/dotnet/diagnostics/blob/master/documentation/dotnet-trace-instructions.md for details. e.g., "Microsoft-DotNETCore-SampleProfiler, gc-verbose, JIT+Contention". - --[JOB].options.traceOutput The name of the trace file. Can be a file prefix (app will add *.DATE*.zip) , or a specific name and no DATE* will be added e.g., c:\traces\mytrace - --[JOB].options.collectCounters Whether to collect dotnet counters. If set and 'counterProviders' is not, System.Runtime will be used by default. - --[JOB].options.counterProviders The name of a performance counter provider from which to collect. e.g., System.Runtime, Microsoft-AspNetCore-Server-Kestrel, Microsoft.AspNetCore.Hosting, Microsoft.AspNetCore.Http.Connections, Grpc.AspNetCore.Server, Grpc.Net.client, Npgsql - --[JOB].collectStartup Whether to include the startup phase in the traces, i.e after the application is launched and before it is marked as ready. For a web application it means before it is ready to accept requests. - --[JOB].collect Whether to collect native traces. Uses PerfView on Windows and Perf/PerfCollect on Linux. - --[JOB].collectArguments Native traces arguments, default is "BufferSizeMB=1024;CircularMB=4096;TplEvents=None;Providers=Microsoft-Diagnostics-DiagnosticSource:0:0;KernelEvents=default+ThreadTime-NetworkTCPIP", other suggested values: "...;GcOnly" - --[JOB].options.dumpType The type of dump to collect. - --[JOB].options.dumpOutput The name of the dump file. Can be a file prefix (app will add *.DATE*.zip) , or a specific name and no DATE* will be added e.g., c:\dumps\mydump - --[JOB].collectDependencies Whether to include the list of project dependencies in the results. - --[JOB].options.downloadOutput Whether to download the job output - --[JOB].options.downloadOutputOutput The name of the output file. Can be a file prefix (app will add *.DATE*.log) , or a specific name and no DATE* will be added e.g., c:\outputs\myoutput - --[JOB].options.downloadBuildLog Whether to download the build log - --[JOB].options.downloadBuildLogOutput The name of the build log file. Can be a file prefix (app will add *.DATE*.log) , or a specific name and no DATE* will be added e.g., c:\builds\mybuild + --[JOB].dotnetTrace Whether to collect a diagnostics trace using dotnet-trace. An optional profile name or list of dotnet-trace + providers can be passed. e.g., true + --[JOB].dotnetTraceProviders A comma-separated list of trace providers. By default the profile 'cpu-sampling' is used. See + https://github.com/dotnet/diagnostics/blob/master/documentation/dotnet-trace-instructions.md for details. + e.g., "Microsoft-DotNETCore-SampleProfiler, gc-verbose, JIT+Contention". + --[JOB].options.traceOutput The name of the trace file. Can be a file prefix (app will add *.DATE*.zip) , or a specific name and no DATE* + will be added e.g., c:\traces\mytrace + --[JOB].options.collectCounters Whether to collect dotnet counters. If set and 'counterProviders' is not, System.Runtime will be used by default. + --[JOB].options.counterProviders The name of a performance counter provider from which to collect. e.g., System.Runtime, + Microsoft-AspNetCore-Server-Kestrel, Microsoft.AspNetCore.Hosting, Microsoft.AspNetCore.Http.Connections, + Grpc.AspNetCore.Server, Grpc.Net.client, Npgsql + --[JOB].collectStartup Whether to include the startup phase in the traces, i.e after the application is launched and before it is marked + as ready. For a web application it means before it is ready to accept requests. + --[JOB].collect Whether to collect native traces. Uses PerfView on Windows and Perf/PerfCollect on Linux. + --[JOB].collectArguments Native traces arguments, default is "BufferSizeMB=1024;CircularMB=4096;TplEvents=None;Providers=Microsoft-Diagnostics-DiagnosticSource:0:0;KernelEvents=default+ThreadTime-NetworkTCPIP", other suggested values: + "...;GcOnly" + --[JOB].options.dumpType The type of dump to collect. + --[JOB].options.dumpOutput The name of the dump file. Can be a file prefix (app will add *.DATE*.zip) , or a specific name and no DATE* will + be added e.g., c:\dumps\mydump + --[JOB].collectDependencies Whether to include the list of project dependencies in the results. + --[JOB].options.downloadOutput Whether to download the job output + --[JOB].options.downloadOutputOutput The name of the output file. Can be a file prefix (app will add *.DATE*.log) , or a specific name and no DATE* will be added e.g., c:\outputs\myoutput + --[JOB].options.downloadBuildLog Whether to download the build log + --[JOB].options.downloadBuildLogOutput The name of the build log file. Can be a file prefix (app will add *.DATE*.log) , or a specific name and no DATE* will be added e.g., c:\builds\mybuild ## Environment - --[JOB].environmentVariables An environment variable key/value pair to assign to the process. Can be used multiple times to define multiple values. - --[JOB].options.requiredOperatingSystem The operating system the job can only run on. - --[JOB].options.requiredArchitecture The architecture the job can only run on. - --[JOB].memoryLimitInBytes The amount of memory available for the process. - --[JOB].cpuSet The list of CPUs available for the process, e.g., "0", "0-3", "1,3-4" - --[JOB].cpuLimitRatio The amount of CPU available for the process, e.g., "0.5". For a 12 cores machines this value would result in 50% out of 1200% total available CPU time. + --[JOB].environmentVariables An environment variable key/value pair to assign to the process. Can be used multiple times to define multiple + values. + --[JOB].options.requiredOperatingSystem The operating system the job can only run on. + --[JOB].options.requiredArchitecture The architecture the job can only run on. + --[JOB].memoryLimitInBytes The amount of memory available for the process. + --[JOB].cpuSet The list of CPUs available for the process, e.g., "0", "0-3", "1,3-4" + --[JOB].cpuLimitRatio The amount of CPU available for the process, e.g., "0.5". For a 12 cores machines this value would result in 50% + out of 1200% total available CPU time. ## Debugging - --[JOB].noClean Whether to keep the work folder on the server or not. Default is false, such that each job is cleaned once it's finished. - --[JOB].options.fetch Whether the benchmark folder is downloaded. e.g., true. For Docker see '--[JOB].source.dockerFetchPath' - --[JOB].options.fetchOutput The name of the fetched archive. Can be a file prefix (app will add *.DATE*.zip) , or a specific name (end in *.zip) and no DATE* will be added e.g., c:\publishedapps\myApp - --[JOB].options.displayOutput Whether to download and display the standard output of the benchmark. - --[JOB].options.displayBuild Whether to download and display the standard output of the build step (works for .NET and Docker). - --[JOB].options.downloadFiles The name of the file(s) to download. The working directory is the published folder. Use '~/' to use the project's location as the base folder. - --[JOB].options.downloadFilesOutput A path where the files will be downloaded. + --[JOB].noClean Whether to keep the work folder on the server or not. Default is false, such that each job is cleaned once it's finished. + --[JOB].options.fetch Whether the benchmark folder is downloaded. e.g., true. For Docker see '--[JOB].dockerFetchPath' + --[JOB].options.fetchOutput The name of the fetched archive. Can be a file prefix (app will add *.DATE*.zip) , or a specific name (end in *.zip) and no DATE* will be added e.g., c:\publishedapps\myApp + --[JOB].options.displayOutput Whether to download and display the standard output of the benchmark. + --[JOB].options.displayBuild Whether to download and display the standard output of the build step (works for .NET and Docker). + --[JOB].options.downloadFiles The name of the file(s) to download. The working directory is the published folder. Use '~/' to use the project's location as the base folder. + --[JOB].options.downloadFilesOutput A path where the files will be downloaded. ## Files - --[JOB].options.buildFiles Build files that will be copied in the project folder before the build occurs. Accepts globing patterns and recursive marker (**). Format is 'path[;destination]'. Path can be a URL. e.g., c:\images\mydockerimage.tar, c:\code\Program.cs. If provided, the destination needs to be a folder name, relative to the project path. - --[JOB].options.outputFiles Output files that will be copied in the published folder after the application is built. Accepts globing patterns and recursive marker (**). Format is 'path[;destination]'. Path can be a URL. e.g., c:\build\Microsoft.AspNetCore.Mvc.dll, c:\files\samples\picture.png;wwwroot\picture.png. If provided, the destination needs to be a folder name, relative to the published path. - --[JOB].options.reuseSource Reuse local or remote sources across benchmarks for the same source. - --[JOB].options.reuseBuild Reuse build files across benchmarks. Don't use with floating runtime versions. - --[JOB].options.beforeScript A command line to execute before the job is started. Current directory is the same as the project or docker file. - --[JOB].options.afterScript A command line to execute after the job is stopped. Current directory is the same as the project or docker file. - --[JOB].options.stoppingScript A command line to execute after the job is stopped. Current directory is the same as the project or docker file. - --[JOB].options.noGitIgnore Whether to ignore the .gitignore file when upload local source or build files. Default is false, meaning the local gitignore file is respected. + --[JOB].options.buildFiles Build files that will be copied in the project folder before the build occurs. Accepts globing patterns and recursive marker (**). Format is 'path[;destination]'. Path can be a URL. e.g., c:\images\mydockerimage.tar, c:\code\Program.cs. If provided, the destination needs to be a folder name, relative to the project path. + --[JOB].options.outputFiles Output files that will be copied in the published folder after the application is built. Accepts globing patterns and recursive marker (**). Format is 'path[;destination]'. Path can be a URL. e.g., c:\build\Microsoft.AspNetCore.Mvc.dll, c:\files\samples\picture.png;wwwroot\picture.png. If provided, the destination needs to be a folder name, relative to the published path. + --[JOB].options.reuseSource Reuse local or remote sources across benchmarks for the same source. + --[JOB].options.reuseBuild Reuse build files across benchmarks. Don't use with floating runtime versions. + --[JOB].options.beforeScript A command line to execute before the job is started. Current directory is the same as the project or docker file. + --[JOB].options.afterScript A command line to execute after the job is stopped. Current directory is the same as the project or docker file. + --[JOB].options.stoppingScript A command line to execute after the job is stopped. Current directory is the same as the project or docker file. + --[JOB].options.noGitIgnore Whether to ignore the .gitignore file when upload local source or build files. Default is false, meaning the local gitignore file is respected. - For script based arguments the following environment variables are available: + For script based arguments the following environment variables are available: - CRANK_PROCESS_ID - CRANK_WORKING_DIRECTORY @@ -138,15 +157,15 @@ Options: Another example using bash to extract some process information and download it from the controller: --application.StoppingScript '/bin/bash -c "cat /proc/$CRANK_PROCESS_ID/smaps > smaps.txt"' --application.downloadFiles smaps.txt - + ## Timeouts - --[JOB].timeout Maximum duration the job in seconds. Defaults to 0 (unlimited). - --[JOB].buildTimeout Maximum duration of the build phase. Defaults to 00:10:00 (10 minutes). - --[JOB].startTimeout Maximum duration of the start phase. Defaults to 00:03:00 (3 minutes). - --[JOB].collectTimeout Maximum duration of the collect phase. Defaults to 00:05:00 (5 minutes). + --[JOB].timeout Maximum duration of the job in seconds. Defaults to 0 (unlimited). + --[JOB].buildTimeout Maximum duration of the build phase. Defaults to 00:10:00 (10 minutes). + --[JOB].startTimeout Maximum duration of the start phase. Defaults to 00:03:00 (3 minutes). + --[JOB].collectTimeout Maximum duration of the collect phase. Defaults to 00:05:00 (5 minutes). ## Measurements - --[JOB].options.discardResults Whether to discard all the results from this job, for instance during a warmup job. + --[JOB].options.discardResults Whether to discard all the results from this job, for instance during a warmup job. diff --git a/src/Microsoft.Crank.Jobs.Bombardier/bombardier.yml b/src/Microsoft.Crank.Jobs.Bombardier/bombardier.yml index 9ed272843d..ab9b7440c4 100644 --- a/src/Microsoft.Crank.Jobs.Bombardier/bombardier.yml +++ b/src/Microsoft.Crank.Jobs.Bombardier/bombardier.yml @@ -9,12 +9,14 @@ jobs: bombardier: - source: - repository: https://github.com/dotnet/crank.git - branchOrCommit: main - project: src/Microsoft.Crank.Jobs.Bombardier/Microsoft.Crank.Jobs.Bombardier.csproj - sourceKey: bombardier - noBuild: true + sources: + crank: + repository: https://github.com/dotnet/crank.git + branchOrCommit: main + sourceKey: bombardier + destinationFolder: '' + project: src/Microsoft.Crank.Jobs.Bombardier/Microsoft.Crank.Jobs.Bombardier.csproj + noBuild: true readyStateText: Bombardier Client waitForExit: true variables: diff --git a/src/Microsoft.Crank.Jobs.H2Load/h2load.yml b/src/Microsoft.Crank.Jobs.H2Load/h2load.yml index 1f0a686468..36dd1ea460 100644 --- a/src/Microsoft.Crank.Jobs.H2Load/h2load.yml +++ b/src/Microsoft.Crank.Jobs.H2Load/h2load.yml @@ -7,12 +7,14 @@ jobs: h2LoadClient: - source: - repository: https://github.com/dotnet/crank.git - branchOrCommit: main - project: src/Microsoft.Crank.Jobs.H2Load/Microsoft.Crank.Jobs.H2Load.csproj - sourceKey: h2load - noBuild: true + sources: + crank: + repository: https://github.com/dotnet/crank.git + branchOrCommit: main + sourceKey: h2load + destinationFolder: '' + project: src/Microsoft.Crank.Jobs.H2Load/Microsoft.Crank.Jobs.H2Load.csproj + noBuild: true readyStateText: H2Load Client waitForExit: true variables: diff --git a/src/Microsoft.Crank.Jobs.HttpClient/httpclient.yml b/src/Microsoft.Crank.Jobs.HttpClient/httpclient.yml index 810587f98e..a2277cad59 100644 --- a/src/Microsoft.Crank.Jobs.HttpClient/httpclient.yml +++ b/src/Microsoft.Crank.Jobs.HttpClient/httpclient.yml @@ -9,10 +9,12 @@ jobs: httpclient: - source: - repository: https://github.com/dotnet/crank.git - branchOrCommit: main - project: src/Microsoft.Crank.Jobs.HttpClient/Microsoft.Crank.Jobs.HttpClient.csproj + sources: + crank: + repository: https://github.com/dotnet/crank.git + branchOrCommit: main + destinationFolder: '' + project: src/Microsoft.Crank.Jobs.HttpClient/Microsoft.Crank.Jobs.HttpClient.csproj readyStateText: Http Client isConsoleApp: true waitForExit: true diff --git a/src/Microsoft.Crank.Jobs.K6/k6.yml b/src/Microsoft.Crank.Jobs.K6/k6.yml index cc5c78a8d3..3d84c217ac 100644 --- a/src/Microsoft.Crank.Jobs.K6/k6.yml +++ b/src/Microsoft.Crank.Jobs.K6/k6.yml @@ -1,11 +1,13 @@ jobs: k6: - source: - repository: https://github.com/dotnet/crank.git - branchOrCommit: main - project: src/Microsoft.Crank.Jobs.K6/Microsoft.Crank.Jobs.K6.csproj - sourceKey: k6 - noBuild: true + sources: + crank: + repository: https://github.com/dotnet/crank.git + branchOrCommit: main + sourceKey: k6 + destinationFolder: '' + project: src/Microsoft.Crank.Jobs.K6/Microsoft.Crank.Jobs.K6.csproj + noBuild: true readyStateText: K6 Client waitForExit: true variables: diff --git a/src/Microsoft.Crank.Jobs.PipeliningClient/pipelining.yml b/src/Microsoft.Crank.Jobs.PipeliningClient/pipelining.yml index fad78bebca..b935b95a9e 100644 --- a/src/Microsoft.Crank.Jobs.PipeliningClient/pipelining.yml +++ b/src/Microsoft.Crank.Jobs.PipeliningClient/pipelining.yml @@ -9,10 +9,12 @@ jobs: pipelining: - source: - repository: https://github.com/dotnet/crank.git - branchOrCommit: main - project: src/Microsoft.Crank.Jobs.PipeliningClient/Microsoft.Crank.Jobs.PipeliningClient.csproj + sources: + crank: + repository: https://github.com/dotnet/crank.git + branchOrCommit: main + destinationFolder: '' + project: src/Microsoft.Crank.Jobs.PipeliningClient/Microsoft.Crank.Jobs.PipeliningClient.csproj readyStateText: Pipelining Client isConsoleApp: true waitForExit: true diff --git a/src/Microsoft.Crank.Jobs.Wrk/wrk.yml b/src/Microsoft.Crank.Jobs.Wrk/wrk.yml index 48c60bfdd5..574d4c7f94 100644 --- a/src/Microsoft.Crank.Jobs.Wrk/wrk.yml +++ b/src/Microsoft.Crank.Jobs.Wrk/wrk.yml @@ -9,12 +9,14 @@ jobs: wrk: - source: - repository: https://github.com/dotnet/crank.git - branchOrCommit: main - project: src/Microsoft.Crank.Jobs.Wrk/Microsoft.Crank.Jobs.Wrk.csproj - sourceKey: wrk - noBuild: true + sources: + crank: + repository: https://github.com/dotnet/crank.git + branchOrCommit: main + sourceKey: wrk + destinationFolder: '' + project: src/Microsoft.Crank.Jobs.Wrk/Microsoft.Crank.Jobs.Wrk.csproj + noBuild: true isConsoleApp: true waitForExit: true variables: diff --git a/src/Microsoft.Crank.Jobs.Wrk2/wrk2.yml b/src/Microsoft.Crank.Jobs.Wrk2/wrk2.yml index d6152eb175..771c179eee 100644 --- a/src/Microsoft.Crank.Jobs.Wrk2/wrk2.yml +++ b/src/Microsoft.Crank.Jobs.Wrk2/wrk2.yml @@ -12,10 +12,12 @@ variables: jobs: wrk2: - source: - repository: https://github.com/dotnet/crank.git - branchOrCommit: main - project: src/Microsoft.Crank.Jobs.Wrk2/Microsoft.Crank.Jobs.Wrk2.csproj + sources: + crank: + repository: https://github.com/dotnet/crank.git + branchOrCommit: main + destinationFolder: '' + project: src/Microsoft.Crank.Jobs.Wrk2/Microsoft.Crank.Jobs.Wrk2.csproj isConsoleApp: true waitForExit: true variables: diff --git a/src/Microsoft.Crank.Models/Job.cs b/src/Microsoft.Crank.Models/Job.cs index fd623c1f66..164fc3cd51 100644 --- a/src/Microsoft.Crank.Models/Job.cs +++ b/src/Microsoft.Crank.Models/Job.cs @@ -78,18 +78,37 @@ public Source Source { get { - if (Sources.Count == 0) + if (Sources.Count == 1) { - return new Source(); - } - else if (Sources.Count == 1) - { - return Sources.Values.Single(); - } - else - { - return null; + var sourceEntry = Sources.Single(); + var source = sourceEntry.Value; + + // Only create a Source object if directly convertible. + if (sourceEntry.Key.Equals(Source.DefaultSource) && string.Empty.Equals(source.DestinationFolder)) + { + return new Source + { + BranchOrCommit = source.BranchOrCommit, + CacheOnAgent = source.CacheOnAgent, + DestinationFolder = source.DestinationFolder, + DockerCommand = DockerCommand, + DockerContextDirectory = DockerContextDirectory, + DockerFetchPath = DockerFetchPath, + DockerFile = DockerFile, + DockerImageName = DockerImageName, + DockerLoad = DockerLoad, + DockerPull = DockerPull, + InitSubmodules = source.InitSubmodules, + LocalFolder = source.LocalFolder, + NoBuild = NoBuild, + Project = Project, + Repository = source.Repository, + SourceKey = source.SourceKey, + SourceCode = source.SourceCode + }; + } } + return null; } set { diff --git a/test/Microsoft.Crank.IntegrationTests/assets/hello.benchmarks.yml b/test/Microsoft.Crank.IntegrationTests/assets/hello.benchmarks.yml index a6e7f2c583..bc39248687 100644 --- a/test/Microsoft.Crank.IntegrationTests/assets/hello.benchmarks.yml +++ b/test/Microsoft.Crank.IntegrationTests/assets/hello.benchmarks.yml @@ -1,14 +1,17 @@ jobs: server: - source: - localFolder: '../hello' - project: hello.csproj + sources: + hello: + localFolder: '../hello' + project: hello/hello.csproj readyStateText: Application started. bombardier: - source: - # uploading the whole source folder since it requires other libraries - localFolder: '../src' - project: Microsoft.Crank.Jobs.Bombardier/Microsoft.Crank.Jobs.Bombardier.csproj + sources: + local: + # uploading the whole source folder since it requires other libraries + localFolder: '../src' + destinationFolder: '' + project: Microsoft.Crank.Jobs.Bombardier/Microsoft.Crank.Jobs.Bombardier.csproj readyStateText: Bombardier Client waitForExit: true variables: diff --git a/test/Microsoft.Crank.IntegrationTests/assets/multiclient.benchmarks.yml b/test/Microsoft.Crank.IntegrationTests/assets/multiclient.benchmarks.yml index 96431938b6..eed787ebe6 100644 --- a/test/Microsoft.Crank.IntegrationTests/assets/multiclient.benchmarks.yml +++ b/test/Microsoft.Crank.IntegrationTests/assets/multiclient.benchmarks.yml @@ -1,14 +1,17 @@ jobs: server: - source: - localFolder: '../hello' - project: hello.csproj + sources: + hello: + localFolder: '../hello' + project: hello/hello.csproj readyStateText: Application started. bombardier: - source: - # uploading the whole source folder since it requires other libraries - localFolder: '../src' - project: Microsoft.Crank.Jobs.Bombardier/Microsoft.Crank.Jobs.Bombardier.csproj + sources: + local: + # uploading the whole source folder since it requires other libraries + localFolder: '../src' + destinationFolder: '' + project: Microsoft.Crank.Jobs.Bombardier/Microsoft.Crank.Jobs.Bombardier.csproj readyStateText: Bombardier Client waitForExit: true variables: diff --git a/test/Microsoft.Crank.IntegrationTests/assets/precommands.benchmarks.yml b/test/Microsoft.Crank.IntegrationTests/assets/precommands.benchmarks.yml index 9ee1f7a4f0..929de4aec6 100644 --- a/test/Microsoft.Crank.IntegrationTests/assets/precommands.benchmarks.yml +++ b/test/Microsoft.Crank.IntegrationTests/assets/precommands.benchmarks.yml @@ -9,10 +9,11 @@ commands: jobs: server: - source: - localFolder: ../../../../hello/Release/net8.0{% if publish %}/{{ rid }}/publish{% endif %} - executable: '{% if publish and rid contains ''win'' %}hello.exe{% elsif publish %}hello{% else %}dotnet{% endif %}' - arguments: '{% if publish == false %}exec hello.dll{% endif %}' + sources: + hello: + localFolder: ../../../../hello/Release/net8.0{% if publish %}/{{ rid }}/publish{% endif %} + executable: '{% if publish and rid contains ''win'' %}hello/hello.exe{% elsif publish %}hello/hello{% else %}dotnet{% endif %}' + arguments: '{% if publish == false %}exec hello/hello.dll{% endif %}' variables: publish: false rid: win-x64 @@ -21,10 +22,12 @@ jobs: - buildHello readyStateText: Application started. bombardier: - source: - # uploading the whole source folder since it requires other libraries - localFolder: '../src' - project: Microsoft.Crank.Jobs.Bombardier/Microsoft.Crank.Jobs.Bombardier.csproj + sources: + local: + # uploading the whole source folder since it requires other libraries + localFolder: '../src' + destinationFolder: '' + project: Microsoft.Crank.Jobs.Bombardier/Microsoft.Crank.Jobs.Bombardier.csproj readyStateText: Bombardier Client waitForExit: true variables: