Add pre and post command support (#666)

* Add pre and post command support

* Don't use C# 12 features

* Address PR comments

* Add unit tests and documentation

* Fix build after merge

* Fix precommands integration test yaml

* Simplify precommands integration test

Co-authored-by: Sébastien Ros <sebastienros@gmail.com>

---------

Co-authored-by: Sébastien Ros <sebastienros@gmail.com>

docs/README.md

@@ -18,6 +18,7 @@
 |**[Using different .NET versions](dotnet_versions.md)** | Benchmarking with different .NET versions.
 |**[Collecting event counters](event_counters.md)** | Collecting predefined and custom event counters.
 |**[Post-processing results](post_processing.md)** | Adding custom results and running scripts.
+|**[Running pre-commands](precommands.md)** | Running commands before the job is pushed to the agent.
 
 
 ## Reference documentation

docs/precommands.md

@@ -0,0 +1,78 @@
+## Description
+
+This tutorial explains how you can define commands that should be run on the local machine prior to a job being sent to the agent.
+
+## Motivation and Use Cases
+
+Crank is capable of building your project on the agent itself, and this is the recommended approach if it is acceptable for you as it means you don't have to worry about cross-platform build concerns. However, sometimes the build itself may take a long time or the job requires uploading a large amount of source files to the agent which may make crank not practical to use. 
+
+With pre-commands, you can define commands to be run on the local machine prior to the job being sent to the agent, Using this, you can build the project first, then upload only the built project to crank instead. This could also be useful if you just want to minimise the amount of data being uploaded to the agent and want to write a pre-command that organises all the data you want into a local folder and using that local folder as the source.
+
+## Example
+
+The file at `/crank/samples/precommands/precommands.benchmarks.yml` demonstrates how to use this.
+
+```yaml
+commands:
+  # Assumes that the crank repository is the working directory 
+  buildHello:
+    - condition: job.environment.platform == "windows"
+      scriptType: batch
+      script: dotnet build -c Release -f net8.0 .\samples\hello\hello.csproj
+    - condition: job.environment.platform != "windows"
+      scriptType: bash
+      script: dotnet build -c Release -f net8.0 ./samples/hello/hello.csproj
+
+jobs:
+  server:
+    source:
+      localFolder: ../../artifacts/bin/hello/Release/net8.0
+    executable: dotnet
+    arguments: hello.dll
+    noBuild: true
+    beforeJob:
+      - buildHello
+    readyStateText: Application started.
+```
+
+In this example, we define a `buildHello` command which will build `hello.csproj` locally depending on the current platform. The `beforeJob` property defines the order of the precommands to be run. The source points to a `localFolder` which is the path to where the built app will be relative to the yaml file. The `executable` and `arguments` properties indicate that the job should run `dotnet hello.dll`, and `noBuild` is set to true to tell the agent that there is no build needed.
+
+## More details
+
+Below is all the options you can set on a command definition:
+
+```yaml
+condition: bool # Defaults to "true". A javascript expression which if evaluated to true, will run the definition.
+script: str # The full script to run, can contain multiple lines.
+filePath: str # A path to a script file.
+scriptType: powershell | bash | batch # The type of script found in the script property or at the filePath.
+continueOnError: bool # Defaults to false. If false, will prevent the job from being sent to the agent if the precommand returns an unsuccessful exit code.
+successExitCodes: [int] # Defaults to [0]. A list of exit codes to be treated as successful in conjunction with continueOnError.
+```
+
+The `condition` property will have access to the whole yaml configuration object via a global variable `configuration`, and access to the current job object via a global variable `job`. An easy way to get information about the local environment is through the `job.environment` property which has the following information:
+
+```yaml
+platform: windows | linux | osx | other
+architecture: x86 | x64 | arm | arm64
+```
+
+`commands` can also be defined inside a `job` if you prefer. When defining a command, you are also able to take advantage of the variable substitution like the rest of the yaml. An example of this is below:
+
+```yaml
+jobs:
+  server:
+    variables:
+      configuration: Release
+      framework: net8.0
+      rid: win-x64
+    commands:
+      publishHello:
+        - condition: job.environment.platform == "windows"
+          scriptType: batch
+          script: dotnet publish -c {{ configuration }} -f {{ framework }} -r {{ rid }} .\samples\hello\hello.csproj
+    beforeJob:
+      - publishHello
+```
+
+In addition to `beforeJob`, you can also specify `afterJob` to run commands locally after the jobs have completed to run any cleanup commands.

samples/precommand/precommand.benchmarks.yml

@@ -0,0 +1,45 @@
+imports:
+  - https://raw.githubusercontent.com/dotnet/crank/main/src/Microsoft.Crank.Jobs.Bombardier/bombardier.yml
+
+commands:
+  # Assumes that the crank repository is the working directory 
+  buildHello:
+    - condition: job.environment.platform == "windows"
+      scriptType: batch
+      script: dotnet build -c Release -f net8.0 .\samples\hello\hello.csproj
+    - condition: job.environment.platform != "windows"
+      scriptType: bash
+      script: dotnet build -c Release -f net8.0 ./samples/hello/hello.csproj
+
+jobs:
+  server:
+    source:
+      localFolder: ../../artifacts/bin/hello/Release/net8.0
+    executable: dotnet
+    arguments: hello.dll
+    noBuild: true
+    beforeJob:
+      - buildHello
+    readyStateText: Application started.
+
+scenarios:
+  hello:
+    application:
+      job: server
+    load:
+      job: bombardier
+      variables:
+        serverPort: 5000
+        path: /
+
+profiles:
+  local:
+    variables:
+      serverAddress: localhost
+    jobs: 
+      application:
+        endpoints: 
+          - http://localhost:5010
+      load:
+        endpoints: 
+          - http://localhost:5010

src/Microsoft.Crank.Agent/Startup.cs

@@ -4447,6 +4447,8 @@ private static async Task<Process> StartProcess(string hostname, string benchmar
                     // we need the full path to run this, as it is not in the path
                     executable = Path.Combine(workingDirectory, executable);
                 }
+
+                commandLine = "";
             }
             else if (job.SelfContained)
             {

src/Microsoft.Crank.Controller/Configuration.cs

@@ -51,6 +51,10 @@ public class Configuration
         /// </summary>
         public List<string> OnResultsCreated { get; set; } = new List<string>();
 
+        /// <summary>
+        /// Definitions of commands to be run on the crank controller machine
+        /// </summary>
+        public Dictionary<string, List<CommandDefinition>> Commands { get; set; } = new();
     }
 
     public class Service

src/Microsoft.Crank.PullRequestBot/ProcessUtil.cs → src/Microsoft.Crank.Controller/ProcessUtil.cs

@@ -10,8 +10,9 @@
 using System.Text;
 using System.Threading;
 using System.Threading.Tasks;
+using Microsoft.Crank.PullRequestBot;
 
-namespace Microsoft.Crank.PullRequestBot
+namespace Microsoft.Crank.Controller
 {
     public static class ProcessUtil
     {
@@ -41,26 +42,25 @@ public static string GetScriptHost()
         private static extern int sys_kill(int pid, int sig);
 
         public static async Task<ProcessResult> RunAsync(
-            string filename, 
-            string arguments, 
-            TimeSpan? timeout = null, 
+            string filename,
+            string arguments,
+            TimeSpan? timeout = null,
             string workingDirectory = null,
-            bool throwOnError = true, 
-            IDictionary<string, string> environmentVariables = null, 
+            bool throwOnError = true,
+            IDictionary<string, string> environmentVariables = null,
             Action<string> outputDataReceived = null,
             bool log = false,
             Action<int> onStart = null,
             Action<int> onStop = null,
             bool captureOutput = false,
             bool captureError = false,
-            CancellationToken cancellationToken = default
-        )
+            CancellationToken cancellationToken = default)
         {
             var logWorkingDirectory = workingDirectory ?? Directory.GetCurrentDirectory();
 
             if (log)
             {
-                Console.WriteLine($"[{logWorkingDirectory}] {filename} {arguments}");
+                Log.Write($"[{logWorkingDirectory}] {filename} {arguments}");
             }
 
             using var process = new Process()
@@ -107,7 +107,7 @@ public static async Task<ProcessResult> RunAsync(
 
                     if (log)
                     {
-                        Console.WriteLine(e.Data);
+                        Log.Write(e.Data);
                     }
                 }
             };
@@ -127,7 +127,7 @@ public static async Task<ProcessResult> RunAsync(
                         outputDataReceived.Invoke(e.Data);
                     }
 
-                    Console.WriteLine("[STDERR] " + e.Data);
+                    Log.WriteError("[STDERR] " + e.Data);
                 }
             };
 
@@ -165,7 +165,7 @@ public static async Task<ProcessResult> RunAsync(
             {
                 if (log)
                 {
-                    Console.WriteLine($"Command '{filename} {arguments}' {(cancellationToken.IsCancellationRequested ? "is canceled" : "timed out")}, process {process.Id} will be terminated");
+                    Log.Write($"Command '{filename} {arguments}' {(cancellationToken.IsCancellationRequested ? "is canceled" : "timed out")}, process {process.Id} will be terminated");
                 }
 
                 if (!RuntimeInformation.IsOSPlatform(OSPlatform.Windows))