Update docs with multiple sources information + fix some bugs (#695)

* Update docs with multiple sources information + fix some bugs

* Port precommand docs to use sources as well.

docs/benchmarking_docker.md

@@ -22,12 +22,13 @@ variables:
 
 jobs:
   server:
-    source:
-      repository: https://github.com/TechEmpower/FrameworkBenchmarks
-      branchOrCommit: master
-      dockerFile: frameworks/Java/netty/netty.dockerfile
-      dockerImageName: netty
-      dockerContextDirectory: frameworks/Java/netty
+    sources:
+      techempower:
+        repository: https://github.com/TechEmpower/FrameworkBenchmarks
+        branchOrCommit: master
+    dockerFile: techempower/frameworks/Java/netty/netty.dockerfile
+    dockerImageName: netty
+    dockerContextDirectory: techempower/frameworks/Java/netty
     port: 8080
 
 scenarios:
@@ -70,25 +71,26 @@ By default it will assume the agent is listening on `http://localhost:5010`. In
 
 ## Understanding the Docker configuration
 
-The `jobs` section in the configuration file defines a job named `server`. It's `source` property contains the information necessary to setup the Docker container. 
+The `jobs` section in the configuration file defines a job named `server`. It's `sources` property contains the information about where to download any sources necessary for the job. In this case there is a single source named `techempower` and so the source will be copied into a directory called `techempower` in the job's working directory. The `dockerFile`, `dockerImageName`, `dockerContextDirectory` are then used to define what docker command is run.
 
 In this case it is a **Netty** application taken from the [TechEmpower](https://github.com/TechEmpower/FrameworkBenchmarks) repository which contains several types of benchmarks for any language and framework. Netty is a Java web application framework.   
 
 ```yml
   server:
-    source:
-      repository: https://github.com/TechEmpower/FrameworkBenchmarks
-      branchOrCommit: master
-      dockerFile: frameworks/Java/netty/netty.dockerfile
-      dockerImageName: netty
-      dockerContextDirectory: frameworks/Java/netty
+    sources:
+      techempower:
+        repository: https://github.com/TechEmpower/FrameworkBenchmarks
+        branchOrCommit: master
+    dockerFile: techempower/frameworks/Java/netty/netty.dockerfile
+    dockerImageName: netty
+    dockerContextDirectory: techempower/frameworks/Java/netty
     port: 8080
 ```
 
-The `dockerFile` property is relative to the root of the repository, and point to the Docker file to build.
+The `dockerFile` property is relative to the job's working directory, and points to the Docker file to build. Note that the path starts with `techempower` because each source is copied to a directory with the name of the source.
 
 The `dockerImageName` is used on the agent to name the container. By setting it Docker will be able to reuse previous cached containers and start benchmarks faster.
 
-The `dockerContextDirectory` is a path representing the working directory inside the Docker file script. Any path using `./` in the Docker file will be this value. It's relative to the root of the source (the clone of the repository).
+The `dockerContextDirectory` is a path representing the working directory inside the Docker file script. Any path using `./` in the Docker file will be this value. It's relative to the job's working directory.
 
 The property `port` is used to let the Crank agent detect when the application is ready to accept requests. When specified, the agent will try to ping the expected endpoint until it successfully answers. Another option is to define the property `readyStateText` with some value that is written on the standard output once the application has started. In the case of a Netty application, the service emits `Httpd started. Listening on: 0.0.0.0/0.0.0.0:8080` when the service is ready. Similarly, an ASP.NET application outputs `Application started. Press Ctrl+C to shut down.`

docs/getting_started.md

@@ -37,10 +37,11 @@ imports:
 
 jobs:
   server:
-    source:
-      repository: https://github.com/dotnet/crank
-      branchOrCommit: main
-      project: samples/hello/hello.csproj
+    sources:
+      crank:
+        repository: https://github.com/dotnet/crank
+        branchOrCommit: main
+    project: crank/samples/hello/hello.csproj
 
 scenarios:
   hello:

docs/local_application.md

@@ -10,32 +10,36 @@ This tutorial explains how to send local source files to an agent instead of clo
 
 ## Define the scenario
 
-In the [Getting Started](getting_started.md) tutorial, the **hello** application is benchmarked. It's `source` section is pointing to the Crank repository to inform the Crank Agent to clone it and build it.
+In the [Getting Started](getting_started.md) tutorial, the **hello** application is benchmarked. It's `sources` section has a source named `crank` which is pointing to the Crank repository to inform the Crank Agent to clone it. By default, a source will be cloned to a directory with the same name as the source. 
+
+The `project` property defines a path to a .NET project file that should be built and run as the work carried out by this job. The project's path starts with `crank` because that is the folder that the source is cloned to, and `samples/hello/hello.csproj` is where the project file is located inside that source.
 
 ```yml
   server:
-    source:
-      repository: https://github.com/dotnet/crank
-      branchOrCommit: main
-      project: samples/hello/hello.csproj
+    sources:
+      crank:
+        repository: https://github.com/dotnet/crank
+        branchOrCommit: main
+    project: crank/samples/hello/hello.csproj
     readyStateText: Application started.
 ```
 
 In cases where you want to iterate quickly on an application, it's easier to do changes locally and send the source file to the agent instead of having it clone changes what you would have to push.
 
-The `source` property of a job has a `localFolder` property that can be set to a local folder. 
+A source inside the `sources` property can specify a `localFolder` property that can be set to a local folder.
 
 The file `/crank/samples/local/local.benchmarks.yml` demonstrates how to use this property instead.
 
 ```yml
   server:
-    source:
-      localFolder: ../../samples/hello
-      project: hello.csproj
+    sources:
+      hello-sample:
+        localFolder: ../../samples/hello
+    project: hello-sample/hello.csproj
     readyStateText: Application started.
 ```
 
-The path is relative to the configuration file that contains it.
+The `localFolder` path is relative to the .yml configuration file that contains it.
 
 Run the following command line to execute this job.
 

docs/microbenchmarks.md

@@ -50,9 +50,10 @@ The following example is available at https://github.com/dotnet/crank/blob/main/
 
 ```yml
   benchmarks:
-    source:
-      localFolder: .
-      project: micro.csproj
+    sources:
+      micro:
+        localFolder: .
+    project: micro/micro.csproj
     variables:
       filterArg: "*"
       jobArg: short

docs/precommands.md

@@ -25,10 +25,11 @@ commands:
 
 jobs:
   server:
-    source:
-      localFolder: ../../artifacts/bin/hello/Release/net8.0
+    sources:
+      hello:
+        localFolder: ../../artifacts/bin/hello/Release/net8.0
     executable: dotnet
-    arguments: hello.dll
+    arguments: hello/hello.dll
     noBuild: true
     beforeJob:
       - buildHello

docs/reference/architecture.md

@@ -16,10 +16,11 @@ imports:
 
 jobs:
   server:
-    source:
-      repository: https://github.com/dotnet/crank
-      branchOrCommit: main
-      project: samples/hello/hello.csproj
+    sources:
+      crank:
+        repository: https://github.com/dotnet/crank
+        branchOrCommit: main
+    project: crank/samples/hello/hello.csproj
 
 scenarios:
   hello:

samples/hello/hello.benchmarks.yml

@@ -3,10 +3,11 @@ imports:
 
 jobs:
   server:
-    source:
-      repository: https://github.com/dotnet/crank
-      branchOrCommit: main
-      project: samples/hello/hello.csproj
+    sources:
+      crank:
+        repository: https://github.com/dotnet/crank
+        branchOrCommit: main
+    project: crank/samples/hello/hello.csproj
     readyStateText: Application started.
 
 scenarios:

samples/local/local.benchmarks.yml

@@ -3,9 +3,10 @@ imports:
 
 jobs:
   server:
-    source:
-      localFolder: ../../samples/hello
-      project: hello.csproj
+    sources:
+      hello-sample:
+        localFolder: ../../samples/hello
+    project: hello-sample/hello.csproj
     readyStateText: Application started.
 
 scenarios:

samples/micro/dotnet.benchmarks.yml

@@ -1,9 +1,10 @@
 jobs:
   dotnet:
-    source:
-      repository: https://github.com/dotnet/performance
-      branchOrCommit: main
-      project: src/benchmarks/micro/MicroBenchmarks.csproj
+    sources:
+      performance:
+        repository: https://github.com/dotnet/performance
+        branchOrCommit: main
+    project: performance/src/benchmarks/micro/MicroBenchmarks.csproj
     variables:
       filterArg: "*"
       jobArg: short

samples/micro/micro.benchmarks.yml

@@ -1,8 +1,9 @@
 jobs:
   benchmarks:
-    source:
-      localFolder: .
-      project: micro.csproj
+    sources:
+      micro:
+        localFolder: .
+    project: micro/micro.csproj
     variables:
       filterArg: "*"
       jobArg: short

samples/netty/netty.benchmarks.yml

@@ -6,12 +6,13 @@ variables:
 
 jobs:
   server:
-    source:
-      repository: https://github.com/TechEmpower/FrameworkBenchmarks
-      branchOrCommit: master
-      dockerFile: frameworks/Java/netty/netty.dockerfile
-      dockerImageName: netty
-      dockerContextDirectory: frameworks/Java/netty
+    sources:
+      techempower:
+        repository: https://github.com/TechEmpower/FrameworkBenchmarks
+        branchOrCommit: master
+    dockerFile: techempower/frameworks/Java/netty/netty.dockerfile
+    dockerImageName: netty
+    dockerContextDirectory: techempower/frameworks/Java/netty
     port: 8080
 
 scenarios:

samples/precommand/precommand.benchmarks.yml

@@ -13,10 +13,11 @@ commands:
 
 jobs:
   server:
-    source:
-      localFolder: ../../artifacts/bin/hello/Release/net8.0
+    sources:
+      hello:
+        localFolder: ../../artifacts/bin/hello/Release/net8.0
     executable: dotnet
-    arguments: hello.dll
+    arguments: hello/hello.dll
     noBuild: true
     beforeJob:
       - buildHello

src/Microsoft.Crank.Controller/Documentation.cs

@@ -11,14 +11,14 @@ These options are specific to a job instance. Replace [JOB] by the name of the s
 
   ## Sources location
 
-  --[JOB].source.repository                                     The git repository containing the source code to build, e.g., https://github.com/aspnet/aspnetcore
-  --[JOB].source.branchOrCommit                                 A branch name or commit hash, e.g., my/branch, my/branch#commit-hash
-  --[JOB].source.initSubmodules                                 Whether to init submodules when a git repository is used, e.g., true
-  --[JOB].source.localFolder                                    The local path containing the source code to upload to the server. e.g., /code/mybenchmarks
+  --[JOB].sources.[SOURCE].repository                           The git repository containing the source code to build, e.g., https://github.com/aspnet/aspnetcore
+  --[JOB].sources.[SOURCE].branchOrCommit                       A branch name or commit hash, e.g., my/branch, my/branch#commit-hash
+  --[JOB].sources.[SOURCE].initSubmodules                       Whether to init submodules when a git repository is used, e.g., true
+  --[JOB].sources.[SOURCE].localFolder                          The local path containing the source code to upload to the server. e.g., /code/mybenchmarks
 
   ## .NET options
 
-  --[JOB].source.project <filename.csproj>                      The project file to build, relative to the source code base path, e.g., src/Benchmarks/Benchmarks.csproj
+  --[JOB].project <filename.csproj>                             The project file to build, relative to the source code base path, e.g., src/Benchmarks/Benchmarks.csproj
   --[JOB].sdkVersion <version>                                  The version of the .NET SDK to install and use. By default the latest available build is used.
   --[JOB].runtimeVersion <version>                              The version of the .NET runtime to install and use. It is defined as MicrosoftNETCoreAppPackageVersion 
                                                                 in the build arguments. By default the latest available build is used. Setting this value forces the app to 
@@ -37,13 +37,13 @@ be deployed as stand-alone.
 
   ## Docker options
 
-  --[JOB].source.dockerFile                                     The local path to the Docker file, e.g., frameworks/Rust/actix/actix-raw.dockerfile
-  --[JOB].source.dockerImageName                                The name of the docker image to create, e.g., actix_raw
-  --[JOB].source.dockerContextDirectory                         The folder in which the Docker file is built relative to, e.g., frameworks/Rust/actix/
-  --[JOB].source.dockerFetchPath                                The path in the Docker container that contains the base path for the --fetch option, e.g., ./output
-  --[JOB].source.dockerLoad                                     The path of an image to use for 'docker load', e.g, ""./myimage.tar""
-  --[JOB].source.dockerPull                                     The image name to pull and run, e.g, ""redis"", ""mcr.microsoft.com/dotnet/aspnet:7.0""
-  --[JOB].source.dockerCommand                                  The 'docker run' command, e.g, ""./startup.sh""
+  --[JOB].dockerFile                                            The local path to the Docker file, e.g., frameworks/Rust/actix/actix-raw.dockerfile
+  --[JOB].dockerImageName                                       The name of the docker image to create, e.g., actix_raw
+  --[JOB].dockerContextDirectory                                The folder in which the Docker file is built relative to, e.g., frameworks/Rust/actix/
+  --[JOB].dockerFetchPath                                       The path in the Docker container that contains the base path for the --fetch option, e.g., ./output
+  --[JOB].dockerLoad                                            The path of an image to use for 'docker load', e.g, ""./myimage.tar""
+  --[JOB].dockerPull                                            The image name to pull and run, e.g, ""redis"", ""mcr.microsoft.com/dotnet/aspnet:7.0""
+  --[JOB].dockerCommand                                         The 'docker run' command, e.g, ""./startup.sh""
   --[JOB].buildArguments <argument>                             An argument to pass to 'docker build' as a '--build-arg' value. Can be used multiple times to define multiple 
                                                                 values.
 
@@ -88,7 +88,7 @@ out of 1200% total available CPU time.
   ## Debugging
 
   --[JOB].noClean <true|false>                                  Whether to keep the work folder on the server or not. Default is false, such that each job is cleaned once it's finished.
-  --[JOB].options.fetch <true|false>                            Whether the benchmark folder is downloaded. e.g., true. For Docker see '--[JOB].source.dockerFetchPath'
+  --[JOB].options.fetch <true|false>                            Whether the benchmark folder is downloaded. e.g., true. For Docker see '--[JOB].dockerFetchPath'
   --[JOB].options.fetchOutput <filename>                        The name of the fetched archive. Can be a file prefix (app will add *.DATE*.zip) , or a specific name (end in *.zip) and no DATE* will be added e.g., c:\publishedapps\myApp
   --[JOB].options.displayOutput <true|false>                    Whether to download and display the standard output of the benchmark.
   --[JOB].options.displayBuild <true|false>                     Whether to download and display the standard output of the build step (works for .NET and Docker).

src/Microsoft.Crank.Controller/Program.cs

@@ -2377,18 +2377,14 @@ public static async Task<JObject> LoadConfigurationAsync(string configurationFil
                         var jobObject = (JObject)job.Value;
                         if (jobObject.ContainsKey("source"))
                         {
-                            var source = (JObject)jobObject["source"];
-                            if (source.ContainsKey("localFolder"))
-                            {
-                                var localFolder = source["localFolder"].ToString();
-
-                                if (!localFolder.StartsWith("http", StringComparison.OrdinalIgnoreCase))
-                                {
-                                    var configurationFilename = new FileInfo(configurationFilenameOrUrl).FullName;
-                                    var resolvedFilename = new FileInfo(Path.Combine(Path.GetDirectoryName(configurationFilename), localFolder)).FullName;
+                            PatchLocalFolderInSource(configurationFilenameOrUrl, (JObject)jobObject["source"]);
+                        }
 
-                                    source["localFolder"] = resolvedFilename;
-                                }
+                        if (jobObject.ContainsKey("sources"))
+                        {
+                            foreach (JProperty source in jobObject["sources"])
+                            {
+                                PatchLocalFolderInSource(configurationFilenameOrUrl, (JObject)source.Value);
                             }
                         }
                     }
@@ -2430,6 +2426,22 @@ public static async Task<JObject> LoadConfigurationAsync(string configurationFil
             }
         }
 
+        private static void PatchLocalFolderInSource(string configurationFilenameOrUrl, JObject source)
+        {
+            if (source.ContainsKey("localFolder"))
+            {
+                var localFolder = source["localFolder"].ToString();
+
+                if (!localFolder.StartsWith("http", StringComparison.OrdinalIgnoreCase))
+                {
+                    var configurationFilename = new FileInfo(configurationFilenameOrUrl).FullName;
+                    var resolvedFilename = new FileInfo(Path.Combine(Path.GetDirectoryName(configurationFilename), localFolder)).FullName;
+
+                    source["localFolder"] = resolvedFilename;
+                }
+            }
+        }
+
         /// <summary>
         /// Merges a JObject into another one.
         /// </summary>