[scripts/ci] CI Test Modified Ports does not fail as expected

[WangWeiLin-MV]

Fix #40124.

Check the issue link, vcpkg ci reported REGRESSION: cyrus-sasl:x64-windows failed with BUILD_FAILED. but the CI task does not emit error.

Fix it by moving step Test Modified Ports from AzureCLI@2 to PowerShell@2.

[WangWeiLin-MV]

See fixes in #40198

---

Information

An earliest abnormal result found so far is Pipelines - Run 20240716.15 logs (visualstudio.com)

A recent normal result failed as expected is Pipelines - Run 20240716.11 logs (visualstudio.com)

According to the test Pipelines - Run 20240731.15 logs (visualstudio.com), all Windows jobs are not failed as expect.

According to the test Pipelines - Run 20240731.20 logs (visualstudio.com), it fails as expected with PowerShell@2.

Possible regression #39896 #39952

Reference

PowerShell@2 - failOnStderr
AzureCLI@2 - failOnStandardError

Test

The CI task test pass with Windows jobs:

• Failed with failure port Pipelines - Run 20240801.4 logs (visualstudio.com)
• vcpkg testing MSVC - no testing

[BillyONeal]

Yeah this is squarely my fault in #39952 -- I saw failures for macOS and Linux and assumed 'surely identical code will have identical behavior everywhere'. I don't understand exactly why failOnStandardError is not working only on Windows, or why my attempts to make more reduced repros to file against the folks who own AzureCLI@2 have not worked.

Either way, propagating the exit code rather than relying on stderr being special is just a good idea in the first place, which shuts this down completely. I've submitted that as an alternative fix in #40198

Thanks for pointing this out!

[BillyONeal]

Thanks again for the analysis! I believe this problem is now fixed.

[WangWeiLin-MV]

Testing

• Test PowerShell@2 failOnStderr
• Test AzureCLI@2 failOnStandardError

Reproduced

#include <windows.h>
#include <iostream>
int main() {
    std::cout << "stdout" << std::endl;
    HANDLE hStderr = ::GetStdHandle(STD_ERROR_HANDLE);
    if (hStderr == INVALID_HANDLE_VALUE) {
      return 1;
    }
    DWORD written = 0;
    const char* msg = "stderr\n";
    if (!::WriteFile(hStderr, msg, strlen(msg), &written, nullptr)) {
        return 1;
    }
    return 0;
}

Reproduced with ::WriteFile, see vcpkg-tool/src/vcpkg/base/messages.cpp#L307-L353.

See microsoft/azure-pipelines-tasks#20291