Make SDL/Guardian scripts and templates flexible

[dagood]

To double check:

• The right tests are in and and the right validation has happened. Guidance: https://github.com/dotnet/core-eng/tree/master/Documentation/Validation

---

Fixes #7592 (see this issue for list of overall changes)

I kicked off a dotnet-release validation branch here (per some instructions in a mail thread--just copied the changes over to a dev branch and requeued an existing build): https://dev.azure.com/dnceng/internal/_build/results?buildId=1233186&view=results

The way I made it more flexible is to make execute-all-sdl-tools.ps1 take an array of object, not just string. It should be backwards compatible with strings, but a hashtable can configure more particulars. For example, I can pass these in the list of tools to get PoliCheck to scan two specific directories:

    @{
      Name="policheck"
      Scenario="eng"
      Args=@(
        "Target < $engDirectory"
      )
    }
    @{
      Name="policheck"
      Scenario=".github"
      Args=@(
        "Target < $dotGitHubDirectory"
      )
    }

I split run-sdl.ps1 into a more focused run-sdl.ps1 and configure-sdl-tool.ps1 for the new "configure everything, then run once" workflow.

[dagood]

https://dev.azure.com/dnceng/internal/_build/results?buildId=1233186&view=results

It looks like my dotnet-release validation build failed early on while downloading some packages. 🙁 (503 (Service Unavailable).) I'd appreciate any help you can provide in testing this, I'm not sure what to expect from this pipeline. /cc @markwilkie

[dagood]

It looks like the 503 errors I'm hitting in the dev branch build and Arcade CI are caused by an ongoing AzDO package feed outage. (Discussion in FR channel.) Will rerun later.

[dagood]

A new build I queued seems to have passed the part that uses the SDL scripts:
https://dev.azure.com/dnceng/internal/_build/results?buildId=1235128&view=results
(The earlier build failed on retry--doesn't seem to support it.)

Do you think there's any more validation that would be reasonable to do here, @adiaaida?

[michellemcdaniel]

I think this is a good amount of validation. I'll take a look at the PR

[michellemcdaniel]

Personally, I think some of the verbose script block calls make the logs a little confusing to read, but I rarely look at the sdl logs anyway, so maybe they are actually helpful when things fail.

Otherwise, this looks good to me.

[dagood]

Personally, I think some of the verbose script block calls make the logs a little confusing to read

Yep, they absolutely do, it's just what I ended up with when I tried implementing this as a little util function. The maintainability/trustability improvement over the widespread code duplication is worth it IMO. I noticed at least one of the duplicate write-host x; & x pairs were incorrect and caused me a problem when I was trying to develop this. 😛