Linker hangs with no output 9/10 times. #88704

hydra · 2021-09-06T20:16:43Z

When building various example projects, I find that the rust-lld.exe linker will just stall instead of creating the .elf file. When it works, it takes about 0.4seconds, most of the time it fails and you have to ctrl+c the command and retry, it's obviously ridiculously frustrating, especially during a pair programming session.

To make sure there was nothing odd on my PC that might be affecting it, I did a complete fresh install of the OS, rust and the tools, yet the same thing still happens.

I was able to record detailed logs using ProcessMonitor of the failure. Attached to this issue are 3 files:

good.CSV, a log from when it works.

good.CSV

The output was this:

PS D:\Users\Hydra\Documents\dev\playground\rust\rtic\rtic-examples\rtic_v6\stm32f4_pwm_monitor> cargo build -j1
   Compiling stm32f4_pwm_input v0.1.0 (D:\Users\Hydra\Documents\dev\playground\rust\rtic\rtic-examples\rtic_v6\stm32f4_pwm_monitor)
    Finished dev [unoptimized + debuginfo] target(s) in 0.70s

bad.CSV and 3) bad-after-killing.CSV. A trace was started and saved as bad.CSV when rust-lld.exe hanged, then a new trace was started, ctrl+c was pressed and the new trace was saved as bad-after-killing.CSV so that you can see what happens before and after ctrl+c was pressed.

bad.CSV
bad-after-killing.CSV

The output, when it fails is this:

...
Compiling stm32f4_pwm_input v0.1.0 (D:\Users\Hydra\Documents\dev\playground\rust\rtic\rtic-examples\rtic_v6\stm32f4_pwm_monitor)
    Building [=========================> ] 98/99: stm32f4_pwm_input(bin)

I'm compiling for an STM32F446RE CPU.

I tried with stable and with nightly

Meta

Project source: rtic-examples.zip

Nightly:

> rustup show
Default host: x86_64-pc-windows-gnu
rustup home:  D:\Users\Hydra\.rustup

installed toolchains
--------------------

stable-x86_64-pc-windows-gnu
nightly-x86_64-pc-windows-gnu (default)

installed targets for active toolchain
--------------------------------------

thumbv7em-none-eabihf
x86_64-pc-windows-gnu

active toolchain
----------------

nightly-x86_64-pc-windows-gnu (default)
rustc 1.57.0-nightly (e30b68353 2021-09-05)

Stable:

> rustup show
Default host: x86_64-pc-windows-gnu
rustup home:  D:\Users\Hydra\.rustup

installed toolchains
--------------------

stable-x86_64-pc-windows-gnu (default)
nightly-x86_64-pc-windows-gnu

installed targets for active toolchain
--------------------------------------

thumbv7em-none-eabihf
x86_64-pc-windows-gnu

active toolchain
----------------

stable-x86_64-pc-windows-gnu (default)
rustc 1.54.0 (a178d0322 2021-07-26)

Hardware:

Processor	AMD Ryzen 9 5950X 16-Core Processor               3.40 GHz
Installed RAM	64.0 GB

OS:

Edition	Windows 10 Pro
Version	21H1
Installed on	‎05/‎09/‎2021
OS build	19043.928
Experience	Windows Feature Experience Pack 120.2212.551.0

If you need more information/logs/etc, just tell me what commands you want me to run and what output/files to provide.

The text was updated successfully, but these errors were encountered:

mati865 · 2021-09-07T07:06:27Z

Could you obtain stacktraces for the hanged process? I though we fixed all hangs like that in LLD 13 but maybe Rust's ancient GCC is even more broken than more recent versions.

hydra · 2021-09-07T15:11:52Z

@mati865 Sure, but how do I obtain a stacktrace for the hung process?

mati865 · 2021-09-07T15:52:50Z

The easiest way is to attach debugger to the process (this sometimes fixes the hang so you cannot get the trace) or create dump by right clicking on the process in task manager and load it into the debugger.
Anyway I'm back home so I've obtained the trace myself:

[0x0]   ntdll!NtWaitForSingleObject + 0x14   
[0x1]   KERNELBASE!WaitForSingleObjectEx + 0x8e   
[0x2]   libwinpthread_1!pthread_cond_init + 0x22c   
[0x3]   libwinpthread_1!pthread_cond_init + 0x36e   
[0x4]   libwinpthread_1!pthread_cond_signal + 0xcf   
[0x5]   rust_lld + 0x282fcd9   
[0x6]   rust_lld + 0x1f6f14c   
[0x7]   rust_lld + 0xdda465   
[0x8]   rust_lld + 0xcc9958   
[0x9]   rust_lld + 0xdb497c   
[0xa]   rust_lld + 0xdc70ab   
[0xb]   rust_lld + 0xdc059a   
[0xc]   rust_lld + 0xe32940   
[0xd]   rust_lld + 0x3cc319   
[0xe]   rust_lld + 0x2b22910   
[0xf]   rust_lld + 0x13f8   
[0x10]   rust_lld + 0x151b   
[0x11]   kernel32!BaseThreadInitThunk + 0x14   
[0x12]   ntdll!RtlUserThreadStart + 0x21

Indeed it looks like bug in old GCC or mingw-w64 used by Rust to build this target.

hydra · 2021-09-07T23:16:12Z

@mati865 Awesome! Thanks ever so much for digging into it, was glad you were able to replicate the issue. Keep us all posted. I and my colleagues will be so happy when this is fixed.

Is it possible to upgrade the tools used by rust to workaround this issue?

mati865 · 2021-09-08T00:34:14Z

Unfortunately there is no easy fix, last time I tried to upgrade the tools it couldn't make it on the CI: #51989
Mingw-builds haven't provided any new build since then, I'll try to provide more up to date toolchain but cannot give any ETA.

I don't know how rust-lld is called here but as a workaround you could try using ld.lld from official LLVM builds or MSYS2.

hydra · 2021-09-08T11:03:08Z

rust-lld is called via cargo -> rustc -> rust-lld IIRC.

Can you reach out the mingw developers to see if they can provide an updated build?

How would I go about using ld.lld ? overwrite the rust-lld.exe with a different .exe or?

hydra · 2021-09-09T17:20:54Z

I found the documentation for changing the linker:

https://doc.rust-lang.org/rustc/codegen-options/index.html#linker-flavor

I will give that a try as time permits and report back.

hydra · 2022-05-24T17:37:56Z

This is still an issue, I changed my linker to the gnu-ld linker for other reasons (map file generation, printing memory usage stats, etc) and forgot about this issue. Today when working on a new project I encountered the error again. I'm using a much newer version of rust since I opened this issue.

rustup show reports the following:

Default host: x86_64-pc-windows-gnu
rustup home:  D:\Users\Hydra\.rustup

installed toolchains

--------------------

stable-x86_64-pc-windows-gnu
nightly-2019-07-01-x86_64-pc-windows-gnu
nightly-2021-08-23-x86_64-pc-windows-gnu
nightly-2021-09-07-x86_64-pc-windows-gnu
nightly-2022-03-27-x86_64-pc-windows-gnu (default)
nightly-2022-04-24-x86_64-pc-windows-gnu
nightly-x86_64-pc-windows-gnu

installed targets for active toolchain
--------------------------------------

thumbv7em-none-eabihf
x86_64-pc-windows-gnu

active toolchain
----------------

nightly-2022-03-27-x86_64-pc-windows-gnu (default)
rustc 1.61.0-nightly (1d9c262ee 2022-03-26)

Has there been any update to the tools? what's the current status of this? are there any other workarounds?

hydra · 2022-07-19T17:34:20Z

Any update on this? I find it a bit crazy that such a fundamental issue, i.e. it randomly fails 9/10 times, hasn't been prioritized over other changes.

hydra · 2022-07-19T17:43:11Z

This still happens on a very recent nightly:

Default host: x86_64-pc-windows-gnu
rustup home:  D:\Users\Hydra\.rustup

installed toolchains
--------------------

nightly-2019-07-01-x86_64-pc-windows-gnu
nightly-2021-08-23-x86_64-pc-windows-gnu
nightly-2021-09-07-x86_64-pc-windows-gnu
nightly-2022-03-27-x86_64-pc-windows-gnu
nightly-2022-04-24-x86_64-pc-windows-gnu
nightly-2022-05-31-x86_64-pc-windows-gnu
nightly-2022-06-17-x86_64-pc-windows-gnu
nightly-2022-07-13-x86_64-pc-windows-gnu (default)

installed targets for active toolchain
--------------------------------------

thumbv7em-none-eabihf
thumbv7m-none-eabi
x86_64-pc-windows-gnu

active toolchain
----------------

nightly-2022-07-13-x86_64-pc-windows-gnu (default)
rustc 1.64.0-nightly (1c7b36d4d 2022-07-12)

example, frustrating, user experience:

D:\Users\Hydra\Documents\dev\playground\rust\rtic\rtic-examples\rtic_v6\stm32f4_pwm_monitor>cargo build
   Compiling stm32f4_pwm_input v0.1.0 (D:\Users\Hydra\Documents\dev\playground\rust\rtic\rtic-examples\rtic_v6\stm32f4_pwm_monitor)
^C  Building [=========================> ] 97/98: stm32f4_pwm_input(bin)
D:\Users\Hydra\Documents\dev\playground\rust\rtic\rtic-examples\rtic_v6\stm32f4_pwm_monitor>cargo build
   Compiling stm32f4_pwm_input v0.1.0 (D:\Users\Hydra\Documents\dev\playground\rust\rtic\rtic-examples\rtic_v6\stm32f4_pwm_monitor)
^C  Building [=========================> ] 97/98: stm32f4_pwm_input(bin)
D:\Users\Hydra\Documents\dev\playground\rust\rtic\rtic-examples\rtic_v6\stm32f4_pwm_monitor>cargo build
   Compiling stm32f4_pwm_input v0.1.0 (D:\Users\Hydra\Documents\dev\playground\rust\rtic\rtic-examples\rtic_v6\stm32f4_pwm_monitor)
^C  Building [=========================> ] 97/98: stm32f4_pwm_input(bin)
D:\Users\Hydra\Documents\dev\playground\rust\rtic\rtic-examples\rtic_v6\stm32f4_pwm_monitor>cargo build
   Compiling stm32f4_pwm_input v0.1.0 (D:\Users\Hydra\Documents\dev\playground\rust\rtic\rtic-examples\rtic_v6\stm32f4_pwm_monitor)
    Finished dev [unoptimized + debuginfo] target(s) in 0.45s

83: Fix parser error on `undef` type in argument list of tail call. r=japaric a=hydra LLVM output was generated by rustc nightly-2022-11-16. I ran into an issue with the parser in the tool and found and 'fixed' the issue. Note: I wasn't able to run the integration tests, they ALL fail on my machine, but I don't think the code change I made is the cause. I have an underlying issue with the rustc linker failing to link targets due to an bug that is fixed, but still unreleased. (See rust-lang/rust#88704) The specific line that fails to parse is line 3 in the example. ``` tail call void %1(i32 noundef 0, i32 undef) #39, !dbg !104902 ``` Specifically, it's missing handling for `undef` after the `i32` type. Co-authored-by: Dominic Clifton <me@dominicclifton.name>

mati865 · 2023-02-17T00:59:23Z

The MinGW version had been updated, does this issue still happen with recent nightly?

hydra · 2023-02-21T11:15:03Z

I'll check with the latest nightly and report back ASAP.

hydra · 2023-06-17T16:34:56Z

The MinGW version had been updated, does this issue still happen with recent nightly?

Hi, Sorry for not getting back to you sooner.

I confirm this doesn't happen with the nightly from 2023-06-17, installed via rustup. To verify I ran rustup default 1.54.0 and the problem occurred again. I again switched back to the new nightly with rustup default nightly-2023-06-17-x86_64-pc-windows-gnu and the problem doesn't occur again.

Happy days!

Do we know exactly which nightly and/or or stable version this issue issue was fixed in so that others coming back to this issue can ensure they are on known-working fixed version?

mati865 · 2023-06-17T19:20:07Z

It was fixed by #100178 because this was GCC/winpthreads bug.

hydra · 2023-06-17T22:23:35Z

ok, closing since it seems to be fixed now.

hydra added the C-bug Category: This is a bug. label Sep 6, 2021

jonas-schievink added the I-hang Issue: The compiler never terminates, due to infinite loops, deadlock, livelock, etc. label Sep 6, 2021

This comment has been minimized.

Sign in to view

rustbot added the O-windows-gnu Toolchain: GNU, Operating system: Windows label Sep 7, 2021

hydra mentioned this issue May 24, 2022

Add --nmagic linker arg, for unaligned flash origin support. rust-embedded/cortex-m-quickstart#95

Merged

mati865 mentioned this issue Aug 26, 2022

Upgrade mingw-w64 on CI #100178

Merged

hydra mentioned this issue Nov 21, 2022

Fix parser error on undef type in argument list of tail call. japaric/cargo-call-stack#83

Merged

hydra mentioned this issue Jun 15, 2023

GNU linker warns “corrupt .drectve” on staticlib binary generated with Rust 1.70 on Windows #112368

Closed

hydra closed this as completed Jun 17, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Linker hangs with no output 9/10 times. #88704

Linker hangs with no output 9/10 times. #88704

hydra commented Sep 6, 2021

mati865 commented Sep 7, 2021

This comment has been minimized.

hydra commented Sep 7, 2021

mati865 commented Sep 7, 2021

hydra commented Sep 7, 2021

mati865 commented Sep 8, 2021

hydra commented Sep 8, 2021

hydra commented Sep 9, 2021

hydra commented May 24, 2022

hydra commented Jul 19, 2022

hydra commented Jul 19, 2022

mati865 commented Feb 17, 2023 •

edited

Loading

hydra commented Feb 21, 2023

hydra commented Jun 17, 2023

mati865 commented Jun 17, 2023

hydra commented Jun 17, 2023

Linker hangs with no output 9/10 times. #88704

Linker hangs with no output 9/10 times. #88704

Comments

hydra commented Sep 6, 2021

Meta

mati865 commented Sep 7, 2021

This comment has been minimized.

hydra commented Sep 7, 2021

mati865 commented Sep 7, 2021

hydra commented Sep 7, 2021

mati865 commented Sep 8, 2021

hydra commented Sep 8, 2021

hydra commented Sep 9, 2021

hydra commented May 24, 2022

hydra commented Jul 19, 2022

hydra commented Jul 19, 2022

mati865 commented Feb 17, 2023 • edited Loading

hydra commented Feb 21, 2023

hydra commented Jun 17, 2023

mati865 commented Jun 17, 2023

hydra commented Jun 17, 2023

mati865 commented Feb 17, 2023 •

edited

Loading