Add pre and post command support

[caaavik-msft]

Summary

This PR adds a feature to crank that allows you to define commands that should be run on the controller machine before and after the run. With this feature, it now becomes possible to write commands to pre-organise or pre-build things on the controller machine before the run starts.

Motivation

The motivation for this was the work towards being able to run benchmarks using crank from the dotnet/performance repository against a local build of the dotnet runtime. Since the runtime is very large, and takes a long time to build, it is better if instead the dotnet runtime is built locally first, and then the build output folder is uploaded to the crank agent and run against that instead. This means that every time before the crank command is run, the developer will need to re-build the runtime each time. With this functionality, it means that we can add pre-commands to crank that will trigger a rebuild of the runtime using all the correct build arguments needed for our scenarios.

Another use case might be that the files that you want to be uploaded to crank may all live in different folders and you don't want to upload all those folders if they contain irrelevant files in them. With this, you could write a pre-command that copies all the files into a temporary directory so that it can be uploaded to the crank agent, then a post-command could then delete that folder afterwards if preferred.

Implementation

An example of a yaml file with commands is below:

commands:
  echoHello:
    - condition: job.environment.platform == "windows"
      shell: batch
      script: echo Hello
      continueOnError: true
    - condition: job.environment.platform != "windows"
      shell: bash
      script: echo "Hello"
      continueOnError: true

jobs:
  myJob:
    variables:
      architecture: x64
    commands:
      build:
        - condition: job.environment.platform == "windows"
          shell: batch
          script: build.cmd --architecture {{ architecture }}
        - condition: job.environment.platform != "windows"
          shell: bash
          script: build.sh --architecture {{ architecture }}
    preCommandOrder:
      - echoHello
      - build
    postCommandOrder:
      - echoHello

Commands are defined under commands either at the root level or at the job level. The command definitions are merged together with the job-level definitions taking precedence over the root-level definitions. Each command has a name, and multiple definitions where each definition has a condition in the form of a javascript expression. Global properties such as job and configuration exist to use in the javascript expression to be able to change the command that is run. If multiple command definitions are true for a given run, the first one that appears in the list will be the one invoked. Also as shown in the sample, it is possible to make use of the job variables inside a command definition.

These are all the properties available inside a command definition:

condition: str # a condition expression to check to see if this definition applies
shell: bash | batch | powershell # what shell to run the script or file in
script: string # the script contents to run
file: string # a path to a file where the script lives
continueOnError: bool # whether or not the run should continue if the script returns an error code (default: false)
successExitCodes: [int] # A list of integers that represent the successful exit codes for use with continueOnError (default: [0])

The PR also adds a new field under Job called Environment which has two fields: Platform which has a value of windows | linux | osx | other and Architecture which has values X86 | X64 | ARM | ARM64. These are useful to use from the conditions which will most likely need to be different for each platform.

Some other design decisions

• I moved ProcessUtil out of the Pull Request Bot into the Controller project since it was useful to implement this feature. The Pull Request Bot already has a reference to the Controller project, but there wasn't really any better place to put common code to be reused among these projects.
• The pre-commands run after the config validation has completed. This means that it's not possible to add a pre-command to start a local crank agent since all the endpoints are validated beforehand.

I've verified all this functionality locally on my machine as well and it works quite well, but I'm open to changing the design here as well if needed.

[sebastienros]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[sebastienros]

Maybe try/catch but just log if there is an error and not break the run.

[caaavik-msft]

Have added one, let me know if it is behaving the way you were thinking

[sebastienros]

Isn't it Cmd (aka command prompt). Batch is the name of the script files.

[caaavik-msft]

I've renamed this to ScriptType since that is a more accurate description than ShellType, this defines what type of script it is

[sebastienros]

I don't like "Pre" and I don't like "Order". Let's assume the array/list is ordered, so not mention that.
In msbuild then have BeforeTarget/AfterTarget, so here I would suggest BeforeJob/AfterJob and assume these are command names without repeating the work command.

[sebastienros]

I am wondering if these should have arguments. After all the scripts could read arguments. These could be a list of name/value properties on each command. So the string[] becomes a Command[] with { name, arguments[] { name, value }

[caaavik-msft]

I've made the name change to BeforeJob and AfterJob. I was thinking about how it would look to add arguments to the commands, but since the commands support variable substitution, I think that variable substitution feels more natural as the approach to parameterisation here. I know at least in my use case for this feature I plan on using > 15 variables and it'd be tedious to have to define all those arguments I think.

[sebastienros]

We'll need some doc/sample to remember how it works.