superpmi: create asm diffs AzDO pipeline

[BruceForstall]

We now have a SuperPMI replay pipeline: #56871.

Based on this work, create a SuperPMI pipeline for generating asm diffs.

The purpose of this pipeline is not to create all the asm diffs and allow the user to download the various asm diffs text files, to avoid doing the (simple) work themselves. It is to use the parallelism available in the AzDO system to quickly determine if there are any asm diffs in any configuration.

This will be used by JIT developers who assert that their change contains no asm diffs, and will be able to quickly show and validate that across our SuperPMI collections.

Note that in contrast to the superpmi-replay pipeline, there is no need here to run with different stress modes.

This pipeline should run automatically for every JIT PR.

The pipeline should create a "summary.md" that summarizes all the asmdiffs run, and upload that to the pipeline artifacts. It should also post it as an "Extensions" page like Antigen/Fuzzlyn do (e.g., https://dev.azure.com/dnceng/public/_build/results?buildId=1452609&view=ms.vss-build-web.run-extensions-tab). A stretch goal would be to post the summary.md results to the GitHub comments stream.

A follow-up goal is to generate base/diff textual .dasm files for a subset of the diffs (perhaps just the "top 20" or so, as reported by the summary.md file) and package these up for easy downloading and viewing. Note that it should be a restricted subset to avoid accidentally generating and uploading .dasm files or every single method context in the case of pervasive diffs.

[ghost]

Tagging subscribers to this area: @JulieLeeMSFT
See info in area-owners.md if you want to be subscribed.

Issue Details

---

We now have a SuperPMI replay pipeline: #56871.

Based on this work, create a SuperPMI pipeline for generating asm diffs.

The purpose of this pipeline is not to create all the asm diffs and allow the user to download the various asm diffs text files, to avoid doing the (simple) work themselves. It is to use the parallelism available in the AzDO system to quickly determine if there are any asm diffs in any configuration.

This will be used by JIT developers who assert that their change contains no asm diffs, and will be able to quickly show and validate that across our SuperPMI collections.

Note that in contrast to the superpmi-replay pipeline, there is no need here to run with different stress modes.

Author:	BruceForstall
Assignees:	-
Labels:	area-CodeGen-coreclr
Milestone:	7.0.0

[kunalspathak]

Few thoughts:

• We need a way to figure out how to determine base commit (should be doable using some git command). Then, we need to do a JIT only build (using jitrolling build) for the baseline JIT.
• Once the asmdiffs are ran, we should generate the summary.md for (at least) CodeSize and PerfScore metrics and upload them as an artifact that a developer can point us to.

[BruceForstall]

If we can figure out the baseline commit (in the upstream main branch), we could just download the JIT rolling build created baseline build. There should be no need to build it ourselves.

[BruceForstall]

One way to figure out the baseline:

The pipelines work in the context of a git clone of the PR branch. With git remote add origin https://github.com/dotnet/runtime (this is there by default),

# The normal AzDO checkout doesn't fetch the 'main' ref, so we need to get that.
git fetch origin main --depth=20
# Where do the merge PR branch (e.g., refs/pull/61179/merge) and the 'main' branch meet?
git merge-base HEAD origin/main
# e.g., outputs f7be57f11b7e0a4782f3836669ec1bebcd53ccd8

The "fetch" commands seem to pass some authorization options, e.g.:

-c http.extraheader="AUTHORIZATION: basic ***"

do we need to do this?

[BruceForstall]

Then we need to walk back the commits to the last JIT one so we can find a commit hash to use for the JIT rolling build.

This is essentially the logic in superpmi.py::process_base_jit_path_arg().

So we probably can't use "--depth=20". Maybe start with pulling down everything for 'main', and prune later if necessary.

[kunalspathak]

So we probably can't use "--depth=20". Maybe start with pulling down everything for 'main', and prune later if necessary.

Right, and when I was experimenting this last time while working on #59598, if we pull everything, it is very expensive in terms of data it pulls down. Perhaps, we will do this only in asmdiff pipeline and not anywhere else, it should be ok though.

[BruceForstall]

Draft PR: #61194

[BruceForstall]

Processing generated .dasm files requires running jit-analyze. This means cloning and building jitutils, using the dotnet CLI. The build machine can cross-build a target-specific, self-contained jit-analyze as follows:

C:\gh\jitutils>dotnet publish -c Release --runtime osx-x64 --self-contained --output c:\gh\jitutils\bin\osx-x64 c:\gh\jitutils\src\jit-analyze\jit-analyze.csproj

or replacing osx-x64 by win-x64, linux-x64, or one of the other appropriate RIDs .

On Windows, you get jit-analyze.exe, on Linux/Mac jit-analyze (which perhaps needs to be made executable on the target machine?)

[kunalspathak]

jit-analyze (which perhaps needs to be made executable on the target machine?)

Possibly, we do that in other pipelines too.

runtime/src/coreclr/scripts/superpmi_benchmarks.py

Line 95 in 1c7100b

def make_executable(file_name):