diff --git a/docs/README.md b/docs/README.md index 670ddff748..e307427d00 100644 --- a/docs/README.md +++ b/docs/README.md @@ -18,6 +18,7 @@ |**[Using different .NET versions](dotnet_versions.md)** | Benchmarking with different .NET versions. |**[Collecting event counters](event_counters.md)** | Collecting predefined and custom event counters. |**[Post-processing results](post_processing.md)** | Adding custom results and running scripts. +|**[Running pre-commands](precommands.md)** | Running commands before the job is pushed to the agent. ## Reference documentation diff --git a/docs/precommands.md b/docs/precommands.md new file mode 100644 index 0000000000..14e6b2f98b --- /dev/null +++ b/docs/precommands.md @@ -0,0 +1,78 @@ +## Description + +This tutorial explains how you can define commands that should be run on the local machine prior to a job being sent to the agent. + +## Motivation and Use Cases + +Crank is capable of building your project on the agent itself, and this is the recommended approach if it is acceptable for you as it means you don't have to worry about cross-platform build concerns. However, sometimes the build itself may take a long time or the job requires uploading a large amount of source files to the agent which may make crank not practical to use. + +With pre-commands, you can define commands to be run on the local machine prior to the job being sent to the agent, Using this, you can build the project first, then upload only the built project to crank instead. This could also be useful if you just want to minimise the amount of data being uploaded to the agent and want to write a pre-command that organises all the data you want into a local folder and using that local folder as the source. + +## Example + +The file at `/crank/samples/precommands/precommands.benchmarks.yml` demonstrates how to use this. + +```yaml +commands: + # Assumes that the crank repository is the working directory + buildHello: + - condition: job.environment.platform == "windows" + scriptType: batch + script: dotnet build -c Release -f net8.0 .\samples\hello\hello.csproj + - condition: job.environment.platform != "windows" + scriptType: bash + script: dotnet build -c Release -f net8.0 ./samples/hello/hello.csproj + +jobs: + server: + source: + localFolder: ../../artifacts/bin/hello/Release/net8.0 + executable: dotnet + arguments: hello.dll + noBuild: true + beforeJob: + - buildHello + readyStateText: Application started. +``` + +In this example, we define a `buildHello` command which will build `hello.csproj` locally depending on the current platform. The `beforeJob` property defines the order of the precommands to be run. The source points to a `localFolder` which is the path to where the built app will be relative to the yaml file. The `executable` and `arguments` properties indicate that the job should run `dotnet hello.dll`, and `noBuild` is set to true to tell the agent that there is no build needed. + +## More details + +Below is all the options you can set on a command definition: + +```yaml +condition: bool # Defaults to "true". A javascript expression which if evaluated to true, will run the definition. +script: str # The full script to run, can contain multiple lines. +filePath: str # A path to a script file. +scriptType: powershell | bash | batch # The type of script found in the script property or at the filePath. +continueOnError: bool # Defaults to false. If false, will prevent the job from being sent to the agent if the precommand returns an unsuccessful exit code. +successExitCodes: [int] # Defaults to [0]. A list of exit codes to be treated as successful in conjunction with continueOnError. +``` + +The `condition` property will have access to the whole yaml configuration object via a global variable `configuration`, and access to the current job object via a global variable `job`. An easy way to get information about the local environment is through the `job.environment` property which has the following information: + +```yaml +platform: windows | linux | osx | other +architecture: x86 | x64 | arm | arm64 +``` + +`commands` can also be defined inside a `job` if you prefer. When defining a command, you are also able to take advantage of the variable substitution like the rest of the yaml. An example of this is below: + +```yaml +jobs: + server: + variables: + configuration: Release + framework: net8.0 + rid: win-x64 + commands: + publishHello: + - condition: job.environment.platform == "windows" + scriptType: batch + script: dotnet publish -c {{ configuration }} -f {{ framework }} -r {{ rid }} .\samples\hello\hello.csproj + beforeJob: + - publishHello +``` + +In addition to `beforeJob`, you can also specify `afterJob` to run commands locally after the jobs have completed to run any cleanup commands. diff --git a/samples/precommand/precommand.benchmarks.yml b/samples/precommand/precommand.benchmarks.yml new file mode 100644 index 0000000000..c5e7a1dd1d --- /dev/null +++ b/samples/precommand/precommand.benchmarks.yml @@ -0,0 +1,45 @@ +imports: + - https://raw.githubusercontent.com/dotnet/crank/main/src/Microsoft.Crank.Jobs.Bombardier/bombardier.yml + +commands: + # Assumes that the crank repository is the working directory + buildHello: + - condition: job.environment.platform == "windows" + scriptType: batch + script: dotnet build -c Release -f net8.0 .\samples\hello\hello.csproj + - condition: job.environment.platform != "windows" + scriptType: bash + script: dotnet build -c Release -f net8.0 ./samples/hello/hello.csproj + +jobs: + server: + source: + localFolder: ../../artifacts/bin/hello/Release/net8.0 + executable: dotnet + arguments: hello.dll + noBuild: true + beforeJob: + - buildHello + readyStateText: Application started. + +scenarios: + hello: + application: + job: server + load: + job: bombardier + variables: + serverPort: 5000 + path: / + +profiles: + local: + variables: + serverAddress: localhost + jobs: + application: + endpoints: + - http://localhost:5010 + load: + endpoints: + - http://localhost:5010 diff --git a/src/Microsoft.Crank.Agent/Startup.cs b/src/Microsoft.Crank.Agent/Startup.cs index 87799cf8f3..33ef71729e 100644 --- a/src/Microsoft.Crank.Agent/Startup.cs +++ b/src/Microsoft.Crank.Agent/Startup.cs @@ -4447,6 +4447,8 @@ private static async Task StartProcess(string hostname, string benchmar // we need the full path to run this, as it is not in the path executable = Path.Combine(workingDirectory, executable); } + + commandLine = ""; } else if (job.SelfContained) { diff --git a/src/Microsoft.Crank.Controller/Configuration.cs b/src/Microsoft.Crank.Controller/Configuration.cs index 5d6778ad7e..18907cd25b 100644 --- a/src/Microsoft.Crank.Controller/Configuration.cs +++ b/src/Microsoft.Crank.Controller/Configuration.cs @@ -51,6 +51,10 @@ public class Configuration /// public List OnResultsCreated { get; set; } = new List(); + /// + /// Definitions of commands to be run on the crank controller machine + /// + public Dictionary> Commands { get; set; } = new(); } public class Service diff --git a/src/Microsoft.Crank.PullRequestBot/ProcessResult.cs b/src/Microsoft.Crank.Controller/ProcessResult.cs similarity index 100% rename from src/Microsoft.Crank.PullRequestBot/ProcessResult.cs rename to src/Microsoft.Crank.Controller/ProcessResult.cs diff --git a/src/Microsoft.Crank.PullRequestBot/ProcessUtil.cs b/src/Microsoft.Crank.Controller/ProcessUtil.cs similarity index 90% rename from src/Microsoft.Crank.PullRequestBot/ProcessUtil.cs rename to src/Microsoft.Crank.Controller/ProcessUtil.cs index 909925351d..39789c44c1 100644 --- a/src/Microsoft.Crank.PullRequestBot/ProcessUtil.cs +++ b/src/Microsoft.Crank.Controller/ProcessUtil.cs @@ -10,8 +10,9 @@ using System.Text; using System.Threading; using System.Threading.Tasks; +using Microsoft.Crank.PullRequestBot; -namespace Microsoft.Crank.PullRequestBot +namespace Microsoft.Crank.Controller { public static class ProcessUtil { @@ -41,26 +42,25 @@ public static string GetScriptHost() private static extern int sys_kill(int pid, int sig); public static async Task RunAsync( - string filename, - string arguments, - TimeSpan? timeout = null, + string filename, + string arguments, + TimeSpan? timeout = null, string workingDirectory = null, - bool throwOnError = true, - IDictionary environmentVariables = null, + bool throwOnError = true, + IDictionary environmentVariables = null, Action outputDataReceived = null, bool log = false, Action onStart = null, Action onStop = null, bool captureOutput = false, bool captureError = false, - CancellationToken cancellationToken = default - ) + CancellationToken cancellationToken = default) { var logWorkingDirectory = workingDirectory ?? Directory.GetCurrentDirectory(); if (log) { - Console.WriteLine($"[{logWorkingDirectory}] {filename} {arguments}"); + Log.Write($"[{logWorkingDirectory}] {filename} {arguments}"); } using var process = new Process() @@ -107,7 +107,7 @@ public static async Task RunAsync( if (log) { - Console.WriteLine(e.Data); + Log.Write(e.Data); } } }; @@ -127,7 +127,7 @@ public static async Task RunAsync( outputDataReceived.Invoke(e.Data); } - Console.WriteLine("[STDERR] " + e.Data); + Log.WriteError("[STDERR] " + e.Data); } }; @@ -165,7 +165,7 @@ public static async Task RunAsync( { if (log) { - Console.WriteLine($"Command '{filename} {arguments}' {(cancellationToken.IsCancellationRequested ? "is canceled" : "timed out")}, process {process.Id} will be terminated"); + Log.Write($"Command '{filename} {arguments}' {(cancellationToken.IsCancellationRequested ? "is canceled" : "timed out")}, process {process.Id} will be terminated"); } if (!RuntimeInformation.IsOSPlatform(OSPlatform.Windows)) diff --git a/src/Microsoft.Crank.Controller/Program.cs b/src/Microsoft.Crank.Controller/Program.cs index 809ff1cbe7..df3b38580b 100644 --- a/src/Microsoft.Crank.Controller/Program.cs +++ b/src/Microsoft.Crank.Controller/Program.cs @@ -5,6 +5,7 @@ using System; using System.Collections.Concurrent; using System.Collections.Generic; +using System.Diagnostics; using System.IO; using System.Linq; using System.Net; @@ -627,6 +628,21 @@ public static int Main(string[] args) VersionChecker.CheckUpdateAsync(_httpClient); #pragma warning restore CS4014 + var engine = new Engine(); + + engine.SetValue("console", _scriptConsole); + engine.SetValue("fs", _scriptFile); + engine.SetValue("configuration", configuration); + + foreach (var dependency in dependencies) + { + var job = configuration.Jobs[dependency]; + if (job.BeforeJob != null && job.BeforeJob.Any()) + { + await RunCommands(job, engine, job.BeforeJob); + } + } + Log.Write($"Running session '{session}' with description '{_descriptionOption.Value()}'"); var isBenchmarkDotNet = dependencies.Any(x => configuration.Jobs[x].Options.BenchmarkDotNet); @@ -663,6 +679,16 @@ public static int Main(string[] args) ); } + foreach (var dependency in dependencies) + { + var job = configuration.Jobs[dependency]; + if (job.AfterJob != null && job.AfterJob.Any()) + { + engine.SetValue("job", job); + await RunCommands(job, engine, job.AfterJob); + } + } + // Display diff if (_compareOption.HasValue()) @@ -1916,6 +1942,7 @@ int interval // Evaluate templates var rootVariables = configuration["Variables"]; + var rootCommands = configuration["Commands"]; foreach (JProperty property in configuration["Jobs"] ?? new JObject()) { @@ -1924,6 +1951,7 @@ int interval var jobVariables = job["Variables"]; var variables = MergeVariables(rootVariables, jobVariables, commandLineVariables); + job["Commands"] = MergeVariables(rootCommands, job["Commands"]); // Apply templates on variables first ApplyTemplates(variables, new TemplateContext(variables.DeepClone())); @@ -2071,6 +2099,140 @@ int interval return result; } + private static async Task RunCommands(Job job, Engine engine, List commands) + { + engine.SetValue("job", job); + foreach (var command in commands) + { + if (!job.Commands.TryGetValue(command, out var definitions)) + { + var availableCommands = String.Join("', '", job.Commands.Keys); + throw new ControllerException($"Could not find a command named '{command}'. Possible values: '{availableCommands}'"); + } + + Log.Write($"Running command '{command}'"); + + var stopwatch = new Stopwatch(); + stopwatch.Start(); + + var foundDefinition = false; + + foreach (var definition in definitions) + { + if (!string.IsNullOrEmpty(definition.Condition)) + { + try + { + var shouldRun = engine.Evaluate(definition.Condition).AsBoolean(); + if (!shouldRun) + continue; + } + catch (Exception ex) + { + Log.WriteError($"Could not evaluate condition [{definition.Condition}], ignoring ..."); + Log.WriteError(ex.ToString()); + } + } + + await RunCommand(command, definition); + foundDefinition = true; + break; + } + + if (!foundDefinition) + throw new ControllerException($"Unable to find valid definition of command '{command}' to run, stopping ..."); + + stopwatch.Stop(); + + Log.Write($"Completed running command '{command}' ... took {stopwatch.Elapsed}"); + } + } + + private static async Task RunCommand(string commandName, CommandDefinition definition) + { + var usesTempFile = false; + var fileName = definition.FilePath; + if (string.IsNullOrEmpty(definition.FilePath)) + { + var extension = definition.ScriptType switch + { + ScriptType.Powershell => ".ps1", + ScriptType.Batch => ".bat", + ScriptType.Bash => ".sh", + _ => null + }; + + fileName = Path.GetFullPath(Path.GetRandomFileName(), Path.GetTempPath()); + if (extension != null) + fileName += extension; + + await File.WriteAllTextAsync(fileName, definition.Script); + + usesTempFile = true; + } + + if (System.OperatingSystem.IsLinux() || System.OperatingSystem.IsMacOS()) + await ProcessUtil.RunAsync("chmod", $"+x {fileName}"); + + string executable; + string arguments; + switch (definition.ScriptType) + { + case ScriptType.Batch: + executable = fileName; + arguments = ""; + break; + case ScriptType.Bash: + executable = "/bin/bash"; + arguments = fileName; + break; + case ScriptType.Powershell: + executable = "powershell.exe"; + arguments = $"-ExecutionPolicy Bypass -NoProfile -NoLogo -File {fileName}"; + break; + default: + throw new ControllerException($"Invalid script type '{definition.ScriptType}' for command '{commandName}'"); + } + + try + { + var result = await ProcessUtil.RunAsync(executable, arguments, log: true); + var successfulExit = false; + foreach (var exitCode in definition.SuccessExitCodes) + { + if (result.ExitCode == exitCode) + { + successfulExit = true; + break; + } + } + + if (!successfulExit) + { + var logMessage = $"Command '{commandName}' returned exit code {result.ExitCode}"; + if (!definition.ContinueOnError) + throw new ControllerException($"{logMessage}, stopping..."); + + Log.WriteError($"{logMessage}, continuing ..."); + } + } + catch (Exception ex) when (ex is not ControllerException) + { + var logMessage = $"Exception was thrown when trying to run command '{commandName}'"; + if (!definition.ContinueOnError) + throw new ControllerException($"{logMessage}, stopping..."); + + Log.WriteError($"{logMessage}, continuing ..."); + } + finally + { + if (usesTempFile && File.Exists(fileName)) + { + File.Delete(fileName); + } + } + } + private static string HashKeyData(T KeyData) { using var sha1 = SHA1.Create(); diff --git a/src/Microsoft.Crank.Models/CommandDefinition.cs b/src/Microsoft.Crank.Models/CommandDefinition.cs new file mode 100644 index 0000000000..d140f7373c --- /dev/null +++ b/src/Microsoft.Crank.Models/CommandDefinition.cs @@ -0,0 +1,18 @@ +// Licensed to the .NET Foundation under one or more agreements. +// The .NET Foundation licenses this file to you under the MIT license. +// See the LICENSE file in the project root for more information. + +using System.Collections.Generic; + +namespace Microsoft.Crank.Models +{ + public class CommandDefinition + { + public string Condition { get; set; } = "true"; + public ScriptType ScriptType { get; set; } = ScriptType.Powershell; + public string Script { get; set; } + public string FilePath { get; set; } + public bool ContinueOnError { get; set; } = false; + public List SuccessExitCodes { get; set; } = new List { 0 }; + } +} diff --git a/src/Microsoft.Crank.Models/EnvironmentData.cs b/src/Microsoft.Crank.Models/EnvironmentData.cs new file mode 100644 index 0000000000..f70fde44be --- /dev/null +++ b/src/Microsoft.Crank.Models/EnvironmentData.cs @@ -0,0 +1,43 @@ +// Licensed to the .NET Foundation under one or more agreements. +// The .NET Foundation licenses this file to you under the MIT license. +// See the LICENSE file in the project root for more information. + +using System; +using System.Runtime.InteropServices; + +namespace Microsoft.Crank.Models +{ + /// + /// A model that stores information about the environment of the current process. + /// This is useful when defining conditions for pre or post commands on the controller. + /// + public class EnvironmentData + { + private static readonly string platform = GetCurrentPlatform(); + private static readonly string architecture = RuntimeInformation.OSArchitecture.ToString(); + + public string Platform => platform; + public string Architecture => architecture; + + private static string GetCurrentPlatform() + { + if (RuntimeInformation.IsOSPlatform(OSPlatform.Windows)) + { + return "windows"; + } + else if (RuntimeInformation.IsOSPlatform(OSPlatform.Linux)) + { + return "linux"; + } + else if (RuntimeInformation.IsOSPlatform(OSPlatform.OSX)) + { + return "osx"; + } + else + { + // Windows, Linux, and OSX are the only platforms that have predefined OSPlatform instances. + return "other"; + } + } + } +} diff --git a/src/Microsoft.Crank.Models/Job.cs b/src/Microsoft.Crank.Models/Job.cs index fec1510957..fd623c1f66 100644 --- a/src/Microsoft.Crank.Models/Job.cs +++ b/src/Microsoft.Crank.Models/Job.cs @@ -283,6 +283,14 @@ public Source Source /// public bool PatchReferences { get; set; } = false; + public Dictionary> Commands { get; set; } + + public List BeforeJob { get; set; } = new(); + + public List AfterJob { get; set; } = new(); + + public EnvironmentData Environment { get; set; } = new EnvironmentData(); + public bool IsDocker() { return !String.IsNullOrEmpty(DockerFile) || !String.IsNullOrEmpty(DockerImageName) || !String.IsNullOrEmpty(DockerPull); diff --git a/src/Microsoft.Crank.Models/ScriptType.cs b/src/Microsoft.Crank.Models/ScriptType.cs new file mode 100644 index 0000000000..eab53cdbc2 --- /dev/null +++ b/src/Microsoft.Crank.Models/ScriptType.cs @@ -0,0 +1,13 @@ +// Licensed to the .NET Foundation under one or more agreements. +// The .NET Foundation licenses this file to you under the MIT license. +// See the LICENSE file in the project root for more information. + +namespace Microsoft.Crank.Models +{ + public enum ScriptType + { + Batch, + Bash, + Powershell, + } +} diff --git a/src/Microsoft.Crank.PullRequestBot/CredentialsHelper.cs b/src/Microsoft.Crank.PullRequestBot/CredentialsHelper.cs index 0a2a42678a..4555855c8d 100644 --- a/src/Microsoft.Crank.PullRequestBot/CredentialsHelper.cs +++ b/src/Microsoft.Crank.PullRequestBot/CredentialsHelper.cs @@ -9,6 +9,7 @@ using Microsoft.IdentityModel.Tokens; using Octokit; using System.Text.RegularExpressions; +using Microsoft.Crank.Controller; namespace Microsoft.Crank.PullRequestBot { diff --git a/src/Microsoft.Crank.PullRequestBot/Program.cs b/src/Microsoft.Crank.PullRequestBot/Program.cs index dfa456d0b7..2f1ac971f8 100644 --- a/src/Microsoft.Crank.PullRequestBot/Program.cs +++ b/src/Microsoft.Crank.PullRequestBot/Program.cs @@ -18,6 +18,7 @@ using System.Threading; using System.Threading.Tasks; using Fluid; +using Microsoft.Crank.Controller; using Newtonsoft.Json; using Newtonsoft.Json.Linq; using Octokit; diff --git a/test/Microsoft.Crank.IntegrationTests/CommonTests.cs b/test/Microsoft.Crank.IntegrationTests/CommonTests.cs index 9f09d500fb..66ec0eff5d 100644 --- a/test/Microsoft.Crank.IntegrationTests/CommonTests.cs +++ b/test/Microsoft.Crank.IntegrationTests/CommonTests.cs @@ -4,6 +4,7 @@ using System; using System.IO; +using System.Runtime.InteropServices; using System.Threading.Tasks; using Microsoft.Crank.Agent; using Xunit; @@ -394,6 +395,54 @@ public async Task BenchmarkHelloWithCpuSetSingleCore() Assert.Contains("ASP.NET Core Version", result.StandardOutput); } + [Fact] + public async Task TestPrecommands() + { + _output.WriteLine($"[TEST] Starting controller"); + + var result = await ProcessUtil.RunAsync( + "dotnet", + $"exec {Path.Combine(_crankDirectory, "crank.dll")} --config ./assets/precommands.benchmarks.yml --scenario hello --profile local", + workingDirectory: _crankTestsDirectory, + captureOutput: true, + timeout: DefaultTimeOut, + throwOnError: false, + outputDataReceived: t => { _output.WriteLine($"[CTL] {t}"); } + ); + + Assert.Equal(0, result.ExitCode); + + Assert.Contains("Requests/sec", result.StandardOutput); + Assert.Contains(".NET Core SDK Version", result.StandardOutput); + Assert.Contains(".NET Runtime Version", result.StandardOutput); + Assert.Contains("ASP.NET Core Version", result.StandardOutput); + } + + [Fact] + public async Task TestPrecommandWithVariables() + { + _output.WriteLine($"[TEST] Starting controller"); + + var rid = RuntimeInformation.RuntimeIdentifier; + + var result = await ProcessUtil.RunAsync( + "dotnet", + $"exec {Path.Combine(_crankDirectory, "crank.dll")} --config ./assets/precommands.benchmarks.yml --scenario hello --profile local --variable publish=true --variable rid={rid}", + workingDirectory: _crankTestsDirectory, + captureOutput: true, + timeout: DefaultTimeOut, + throwOnError: false, + outputDataReceived: t => { _output.WriteLine($"[CTL] {t}"); } + ); + + Assert.Equal(0, result.ExitCode); + + Assert.Contains("Requests/sec", result.StandardOutput); + Assert.Contains(".NET Core SDK Version", result.StandardOutput); + Assert.Contains(".NET Runtime Version", result.StandardOutput); + Assert.Contains("ASP.NET Core Version", result.StandardOutput); + } + public void Dispose() { _output.WriteLine(_agent.FlushOutput()); diff --git a/test/Microsoft.Crank.IntegrationTests/assets/precommands.benchmarks.yml b/test/Microsoft.Crank.IntegrationTests/assets/precommands.benchmarks.yml new file mode 100644 index 0000000000..9ee1f7a4f0 --- /dev/null +++ b/test/Microsoft.Crank.IntegrationTests/assets/precommands.benchmarks.yml @@ -0,0 +1,56 @@ +commands: + buildHello: + - condition: job.environment.platform == "windows" + scriptType: batch + script: dotnet {% if publish %}publish -r {{ rid }}{% else %}build{% endif %} -c Release -f net8.0 .\hello\hello.csproj + - condition: job.environment.platform != "windows" + scriptType: bash + script: dotnet {% if publish %}publish -r {{ rid }}{% else %}build{% endif %} -c Release -f net8.0 ./hello/hello.csproj + +jobs: + server: + source: + localFolder: ../../../../hello/Release/net8.0{% if publish %}/{{ rid }}/publish{% endif %} + executable: '{% if publish and rid contains ''win'' %}hello.exe{% elsif publish %}hello{% else %}dotnet{% endif %}' + arguments: '{% if publish == false %}exec hello.dll{% endif %}' + variables: + publish: false + rid: win-x64 + noBuild: true + beforeJob: + - buildHello + readyStateText: Application started. + bombardier: + source: + # uploading the whole source folder since it requires other libraries + localFolder: '../src' + project: Microsoft.Crank.Jobs.Bombardier/Microsoft.Crank.Jobs.Bombardier.csproj + readyStateText: Bombardier Client + waitForExit: true + variables: + serverScheme: http + serverAddress: localhost + serverPort: 5000 + arguments: "-c 1 -w 3 -d 3 --insecure -l --fasthttp {{serverScheme}}://{{serverAddress}}:{{serverPort}}{{path}}" + +scenarios: + hello: + application: + job: server + load: + job: bombardier + variables: + path: / + +profiles: + local: + variables: + serverPort: 5000 + serverAddress: localhost + jobs: + application: + endpoints: + - http://localhost:5010 + load: + endpoints: + - http://localhost:5010