JIT rolling build can build and upload non-JIT changes

[BruceForstall]

As described here:

The jitrollingbuild pipeline is set to build every time the JIT changes, and not batch changes:

trigger:
  batch: false
  branches:
    include:
    - main
  paths:
    include:
    - src/coreclr/jit/*
    - src/coreclr/inc/jiteeversionguid.h

However, it appears that AzDO doesn't kick off the build immediately; I noticed a delay of about 10 minutes for this particular change. In a few previous cases for JIT changes, there were also delays before the AzDO job kicked off, and in those cases, another change was merged before the kickoff, and AzDO built that change, not the JIT change that kicked off the job. Because of this, when we upload the JIT to Azure Storage, we use a git hash of a non-JIT change, and when superpmi.py looks for a baseline JIT, it doesn't find the hash of the JIT that should have been built.

Summary: there is an assumption that the pipeline builds every JIT change at the git hash of the JIT change. That doesn't appear to be the case with AzDO. I'm not sure if there's a config to force that to happen. E.g., what if 2 JIT changes were merged at almost exactly the same time? AzDO should build both of them, but does it?

Possibly we could change the jitrollingbuild.py upload script to walk back the git log for the built change and use the first hash that actually has JIT changes in it, to work around this problem.

[ghost]

Tagging subscribers to this area: @JulieLeeMSFT
See info in area-owners.md if you want to be subscribed.

Issue Details

---

As described here:

The jitrollingbuild pipeline is set to build every time the JIT changes, and not batch changes:

trigger:
  batch: false
  branches:
    include:
    - main
  paths:
    include:
    - src/coreclr/jit/*
    - src/coreclr/inc/jiteeversionguid.h

However, it appears that AzDO doesn't kick off the build immediately; I noticed a delay of about 10 minutes for this particular change. In a few previous cases for JIT changes, there were also delays before the AzDO job kicked off, and in those cases, another change was merged before the kickoff, and AzDO built that change, not the JIT change that kicked off the job. Because of this, when we upload the JIT to Azure Storage, we use a git hash of a non-JIT change, and when superpmi.py looks for a baseline JIT, it doesn't find the hash of the JIT that should have been built.

Summary: there is an assumption that the pipeline builds every JIT change at the git hash of the JIT change. That doesn't appear to be the case with AzDO. I'm not sure if there's a config to force that to happen. E.g., what if 2 JIT changes were merged at almost exactly the same time? AzDO should build both of them, but does it?

Possibly we could change the jitrollingbuild.py upload script to walk back the git log for the built change and use the first hash that actually has JIT changes in it, to work around this problem.

Author:	BruceForstall
Assignees:	-
Labels:	area-CodeGen-coreclr
Milestone:	7.0.0

[BruceForstall]

@MattGal @hoyosjs @safern Do any of you understand how AzDO triggering works? In particular, why it triggers a build on a git hash after the thing I triggered.

[MattGal]

@MattGal @hoyosjs @safern Do any of you understand how AzDO triggering works? In particular, why it triggers a build on a git hash after the thing I triggered.

Apologies, I started parsing that sentence and got lost. In most cases though I believe it's just a webhook or equivalent from GH to AzDO, which sometimes can fail to be delivered, get multi-delivered, or be delayed. Can you give examples on the build that got triggered that you're curious about?

[BruceForstall]

One example was:

https://dev.azure.com/dnceng/internal/_build/results?buildId=1577232&view=results

The triggering merge was https://dev.azure.com/dnceng/internal/_git/dotnet-runtime/commit/d55a1dfa4e39a26a8ea8d161f733c72805f31f7f at 4:18pm.

There was a closely following merge https://dev.azure.com/dnceng/internal/_git/dotnet-runtime/commit/9eb90511ff6d079fa02832936c693d36289fc02b at 4:25pm.

The pipeline didn't start running until 4:36pm, and picked up the change at 4:25pm.

We had code that depended on the git hash being built by the pipeline being the same as the triggering merge, but this wasn't true.

I added some compensating code in #64480 that will look backwards in the git history on the build to find the triggering change (specifically, to find the most recent change to the JIT source directory; this needs to match the YML file "trigger" path to be fully correct).

[MattGal]

This surely does seem like a bug in the way non-batched pipeline triggers are working, so I started an email thread with the pipelines PM and I'll update back here if anything interesting falls out of that conversation.