ci(windows): fix CI failure on x86_64-pc-windows-gnu

[rami3l]

Currently, the master CI fails on x86_64-pc-windows-gnu on

error: linking with `x86_64-w64-mingw32-gcc` failed: exit code: 1
  |
  = note: "x86_64-w64-mingw32-gcc" "-fno-use-linker-plugin" "-Wl,--dynamicbase" "-Wl,--disable-auto-image-base" "-m64" "-Wl,--high-entropy-va" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-gnu\\lib\\rsbegin.o" "C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\rustcvO6rlT\\symbols.o" "D:\\a\\rustup\\rustup\\target\\x86_64-pc-windows-gnu\\release\\deps\\rustup_init-2e95520dda2997b5.rustup_init.61fa2ea700060ea9-cgu.0.rcgu.o" "-L" "D:\\a\\rustup\\rustup\\target\\x86_64-pc-windows-gnu\\release\\deps" "-L" "D:\\a\\rustup\\rustup\\target\\release\\deps" "-L" "C:\\Users\\runneradmin\\.cargo\\registry\\src\\index.crates.io-6f17d22bba15001f\\windows_x86_64_gnu-0.52.5\\lib" "-L" "C:\\Users\\runneradmin\\.cargo\\registry\\src\\index.crates.io-6f17d22bba15001f\\windows_x86_64_gnu-0.48.5\\lib" "-L" "D:\\a\\rustup\\rustup\\target\\x86_64-pc-windows-gnu\\release\\build\\curl-sys-8fe5b342eab26db9\\out\\build" "-L" "D:\\a\\rustup\\rustup\\target\\x86_64-pc-windows-gnu\\release\\build\\libz-sys-3694b72ddf7bf6f5\\out\\lib" "-L" "D:\\a\\rustup\\rustup\\target\\x86_64-pc-windows-gnu\\release\\build\\libz-sys-3694b72ddf7bf6f5\\out\\lib" "-L" "D:\\a\\rustup\\rustup\\target\\x86_64-pc-windows-gnu\\release\\build\\ring-da2b99278300f509\\out" "-L" "D:\\a\\rustup\\rustup\\target\\x86_64-pc-windows-gnu\\release\\build\\sys-info-cd5759db882db8d7\\out" "-L" "C:\\Users\\runneradmin\\.cargo\\registry\\src\\index.crates.io-6f17d22bba15001f\\winapi-x86_64-pc-windows-gnu-0.4.0\\lib" "-L" "C:\\Users\\runneradmin\\.cargo\\registry\\src\\index.crates.io-6f17d22bba15001f\\windows_x86_64_gnu-0.42.2\\lib" "-L" "D:\\a\\rustup\\rustup\\target\\x86_64-pc-windows-gnu\\release\\build\\lzma-sys-33f2e3f950e15d22\\out" "-L" "D:\\a\\rustup\\rustup\\target\\x86_64-pc-windows-gnu\\release\\build\\zstd-sys-efab3e8ff7683f87\\out" "-L" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-gnu\\lib" "-Wl,-Bstatic" "C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\rustcvO6rlT\\libzstd_sys-0366df098ca69c70.rlib" "C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\rustcvO6rlT\\liblzma_sys-8c956afe6bdd1895.rlib" "C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\rustcvO6rlT\\libsys_info-bedaaef05679663a.rlib" "C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\rustcvO6rlT\\libring-249b4006c64c178b.rlib" "C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\rustcvO6rlT\\libcurl_sys-2691bd68069c2e05.rlib" "C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\rustcvO6rlT\\liblibz_sys-b6ddfb1e1e5ff67a.rlib" "C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\rustcvO6rlT\\libstd-a1f74822451877d1.rlib" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-gnu\\lib\\libcompiler_builtins-b44a9797859024b2.rlib" "-Wl,-Bdynamic" "-lpsapi" "-lpowrprof" "-lwinapi_advapi32" "-lwinapi_cfgmgr32" "-lwinapi_gdi32" "-lwinapi_kernel32" "-lwinapi_msimg32" "-lwinapi_ole32" "-lwinapi_opengl32" "-lwinapi_psapi" "-lwinapi_shell32" "-lwinapi_synchronization" "-lwinapi_user32" "-lwinapi_userenv" "-lwinapi_winspool" "-lwindows" "-lws2_32" "-lcrypt32" "-lbcrypt" "-ladvapi32" "-ladvapi32" "-lkernel32" "-lole32" "-loleaut32" "-lwindows.0.52.0" "-lntdll" "-lwindows.0.48.5" "-lkernel32" "-ladvapi32" "-lkernel32" "-lntdll" "-luserenv" "-lws2_32" "-lkernel32" "-lws2_32" "-lkernel32" "-lgcc_eh" "-l:libpthread.a" "-lmsvcrt" "-lmingwex" "-lmingw32" "-lgcc" "-lmsvcrt" "-luser32" "-lkernel32" "-Wl,--nxcompat" "-L" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-gnu\\lib" "-o" "D:\\a\\rustup\\rustup\\target\\x86_64-pc-windows-gnu\\release\\deps\\rustup_init-2e95520dda2997b5.exe" "-Wl,--gc-sections" "-no-pie" "-Wl,-O1" "-nodefaultlibs" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-gnu\\lib\\rsend.o"
  = note: Warning: corrupt .drectve at end of def file
          Warning: corrupt .drectve at end of def file
          collect2.exe: error: ld returned 5 exit status

This PR tries to address this issue while moving away from the previously hardcoded MinGW version (6.3.0 from 2017) at the same time.

Concerns

• Should we keep the x86_64-pc-windows-gnu CI enabled even on PRs? I guess we should, at least for the moment.

[djc]

Thanks for picking this up!

[rami3l]

@djc Sorry this PR is not ready yet. I thought it was fixed but obviously I wasn't looking at the right direction 😓

@mati865 do you happen to have any idea on this one? The context is that our Windows/GNU CI has been failing since merging #3367 and we have now idea why this is the case 🙏
I've tried @ChrisDenton's suggestions such as upgrading MinGW, installing *-gnu as the default host, disabling LTO, etc. but nothing seems to be working so far...

[mati865]

Looks like a linker crash to me. LTO is unrelated because Rust doesn't use linker's LTO (unless you explicitly enable it: https://doc.rust-lang.org/rustc/linker-plugin-lto.html). I can only guess what is crashing it but I'd bet gc-sections. In that case RUSTFLAGS="-C link-arg=-Wl,--no-gc-sections" could help.
Another workaround would be to use LLD.

[rami3l]

Looks like a linker crash to me. LTO is unrelated because Rust doesn't use linker's LTO (unless you explicitly enable it: https://doc.rust-lang.org/rustc/linker-plugin-lto.html). I can only guess what is crashing it but I'd bet gc-sections. In that case RUSTFLAGS="-C link-arg=-Wl,--no-gc-sections" could help. Another workaround would be to use LLD.

@mati865 Thanks for helping out!

Unfortunately, now the error is different, but the build is still a failure, am I doing anything wrong?

  = note: Warning: corrupt .drectve at end of def file
          Warning: corrupt .drectve at end of def file
          C:/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: warning: C:\Users\runneradmin\.cargo\registry\src\index.crates.io-6f17d22bba15001f\windows_x86_64_gnu-0.42.2\lib/libwindows.a(ntdll_dll_s00014.o): local symbol `' has no section
          collect2.exe: error: ld returned 1 exit status

[mati865]

I have no idea what it means but maybe this section causes the crash with --gc-sections, or it's entirely unrelated...
Hopefully LLD would help, I'll see soon if I can reproduce it locally.

[rami3l]

@mati865 I just tried with ld.lld.exe/rust-lld.exe and it gives another kind of error. Looks like an include path problem. Maybe that's easier for you than the previous one?

error: linking with `rust-lld.exe` failed: exit code: 1
  |
  = note: "rust-lld.exe" "-flavor" "gnu" "--dynamicbase" "--disable-auto-image-base" "-m" "i386pep" "--high-entropy-va" "C:\\Users\\runneradmin\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-gnu\\lib\\rsbegin.o" "C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\rustcVUMshG\\symbols.o" "D:\\a\\rustup\\rustup\\target\\x86_64-pc-windows-gnu\\debug\\deps\\rustup_init-02e6c105cc009354.14p56amfjwwsn5jm.rcgu.o" "D:\\a\\rustup\\rustup\\target\\x86_64-pc-windows-gnu\\debug\\deps\\rustup_init-02e6c105cc009354.17ib23qhpn1v4l6i.rcgu.o" "D:\\a\\rustup\\rustup\\target\\x86_64-pc-windows-gnu\\debug\\deps\\rustup_init-02e6c105cc009354.19e13oqkyvw78abj.rcgu.o" "D:\\a\\rustup\\rustup\\target\\x86_64-pc-windows-gnu\\debug\\deps\\rustup_init-02e6c105cc009354.1d01k4kqg3wx0lrc.rcgu.o" "D:\\a\\rustup\\rustup\\target\\x86_64-pc-windows-gnu\\debug\\deps\\rustup_init-02e6c105cc009354.1eteqsfeqh29lh61.rcgu.o" "D:\\a\\rustup\\rustup\\target\\x86_64-pc-windows-gnu\\debug\\deps\\rustup_init-02e6c105cc009354
  = note: lld: error: unable to find library -lpsapi
          lld: error: unable to find library -lpowrprof
          lld: error: unable to find library -lws2_32
          lld: error: unable to find library -lcrypt32
          lld: error: unable to find library -lbcrypt
          lld: error: unable to find library -ladvapi32
          lld: error: unable to find library -ladvapi32
          lld: error: unable to find library -lkernel32
          lld: error: unable to find library -lole32
          lld: error: unable to find library -loleaut32
          lld: error: unable to find library -lntdll
          lld: error: unable to find library -lkernel32
          lld: error: unable to find library -ladvapi32
          lld: error: unable to find library -lkernel32
          lld: error: unable to find library -lntdll
          lld: error: unable to find library -luserenv
          lld: error: unable to find library -lws2_32
          lld: error: unable to find library -lkernel32
          lld: error: unable to find library -lws2_32
          lld: error: unable to find library -lkernel32
          lld: error: too many errors emitted, stopping now

[mati865]

@rami3l LLD should be used like on Unix systems, via cc/gcc/clang. So the easiest way would be to use -fuse-ld=lld. No idea why it didn't work previously.

[rami3l]

@rami3l LLD should be used like on Unix systems, via cc/gcc/clang. So the easiest way would be to use -fuse-ld=lld. No idea why it didn't work previously.

Yes, I did exactly that in https://github.com/rust-lang/rustup/actions/runs/9321950575/job/25661988047, but that... still looks like a GNU ld error?

[mati865]

Yeah, for some reason LLD was not used.

Locally running with MSYS2 (ucrt64 env) I got slightly different error couple time:

error: could not exec the linker `x86_64-w64-mingw32-gcc`
  |
  = note: Permission denied. (os error 5)
...

but all new builds since then do pass (doing cargo clean before each), so I cannot reliably reproduce it with recent Binutils.

Upon trying https://github.com/niXman/mingw-builds-binaries/releases/tag/13.2.0-rt_v11-rev0 that you install via chocolatey it seems to reproduce every time.

Also LLD didn't help, there is probably some incompatibility in how GCC and Clang generate the sections:

$ RUSTFLAGS=-Clink-arg=-fuse-ld=lld cargo b -r
...
error: linking with `x86_64-w64-mingw32-gcc` failed: exit code: 1
  |
  = note: "x86_64-w64-mingw32-gcc" ...
  = note: ld.lld: error: undefined symbol: __declspec(dllimport) _sys_nerr
          >>> referenced by libcurl_sys-5f60248bcb1d72ed.rlib(4da0da058b5259b3-strerror.o):(.text$Curl_strerror)

          ld.lld: error: undefined symbol: __declspec(dllimport) _sys_errlist
          >>> referenced by libcurl_sys-5f60248bcb1d72ed.rlib(4da0da058b5259b3-strerror.o):(.text$Curl_strerror)
          collect2.exe: error: ld returned 1 exit status

So the solution here would be to use more recent tools, for an example using MSYS2 but anything modern should work.
MSYS2 has GH Action available so this should do the trick:

  - uses: msys2/setup-msys2@v2
    with:
      msystem: MINGW64

[rami3l]

I suspect we've been using the bundled (sigh) mingw all along (!) and have encountered something similar to actions/runner-images#5459.

... and the version so far is:

    "mingw": {
        "version": "12.2.0",
        "runtime": "ucrt"
    },

https://github.com/actions/runner-images/blob/25c57a1bbfaa5fbffc2e221b4015754e3815f7c5/images/windows/toolsets/toolset-2022.json#L154-L157

... which explains everything.

However, no matter how I install msys2 and adjust the PATH, I'll always get the following:

> Get-Command x86_64-w64-mingw32-gcc

CommandType     Name                                               Version    Source                                   
-----------     ----                                               -------    ------                                   
Application     x86_64-w64-mingw32-gcc.exe                         0.0.0.0    C:\mingw64\bin\x86_64-w64-mingw32-gcc.exe

Update: After trying another installer, it finally worked! Thanks a lot for your pointers @mati865 and @ChrisDenton!