15.0.7 runtime build fails on armv7l with missing unwinder #60115
Comments
|
I see a lot of linker errors, probably the system linker is not up to the task: /usr/bin/ld: /usr/bin/ld: DWARF error: invalid or unhandled FORM value: 0x25The linker is 2.35.2: $ /usr/bin/ld --version
GNU ld (GNU Binutils for Raspbian) 2.35.2 |
|
https://sourceware.org/bugzilla/show_bug.cgi?id=27521 This? LLVM states: |
|
The DWARF error seems to be the same, but I'm not sure that the undefined references are caused by it. The linker line explicitly asks for |
|
I checked and in my production build environment I use gcc 12 & binutils 2.39; I don't see the DWARF error, but the arm 32-bit build of the runtime still fails, so I would not exclude other problems than binutils yet. |
|
I found this commit that changes some details of runtime builds: There is also a comment mentioning a reorder of the runtimes, to fix some dependency issues, with |
|
@llvm/issue-subscribers-backend-arm |
|
I tried a new build with a different order in LLVM_ENABLE_RUNTIMES, but it did not fix the problem... |
|
We (Linaro) saw this previously with scudo - #56900 And I have hit it with GWPASAN like you have, but applying the same sort of change didn't fix anything. At this time I don't understand exactly how the fix for scudo worked. |
|
Hi David @DavidSpickett, thank you for confirming this bug. I'll postpone my 15.x release and stick to 14.x till the problem is identified and fixed. |
|
Do we know of other Linux 32-bit arm distributions which include llvm/clang 15, to check if they applied any patches? |
ilg-ul
changed the title
|
@ilg-ul I am also a happy clang-14 user on 32 bit arm7l so information on this topic is very relevant to me too. |
|
Let's hope that the 32-bit clang can still be fixed. |
|
Can someone try a build similar to the one presented in the beginning of this topic, but with the most recent 16.x, to confirm that the problem is still present? (my berries are pretty busy these days). |
|
I will try to reproduce and confirm where this started.
For this see https://discourse.llvm.org/t/bin-ld-dwarf-error-invalid-or-unhandled-form-value-0x25/67620/2. You are right that a new linker is needed, or fall back to dwarf 4. |
|
I asked a friend to run a build with the latest master, to be sure the problem is still there; we'll know it in about 8 hours...
I don't think that the dwarf errors are relevant for this. |
One small correction, my build script for the 14.x release did not fully pass, I had to disable some features, so the problem might be a bit older. But I think that the goal is to be sure the master branch passes the build; for the 15.x releases we can apply a patch, if necessary. |
|
I have a workaround for the time being. Apply these two patches: If you get conflicts doing so I can prepare some versions that apply cleanly to the 15 branch. Then add You will have to modify the I trimmed down your initial command to save some time but here is what I used, for reference: We will look for the root cause of the issue and I will work to get those patches landed to improve the situation for llvm 16. |
|
Thank you, David. Applying the patches went without conflicts, just a warning related to an offset of 14 lines; I guess it is not a problem. I started a new fresh build using the following commands: rm -rf ${HOME}/Work/llvm-test/build/*
cmake \
--fresh \
-S "${HOME}/Work/llvm-test/sources/llvm-project-15.0.7.src/llvm/" \
-B "${HOME}/Work/llvm-test/build/" \
-G Ninja \
-DCMAKE_INSTALL_PREFIX="${HOME}/Work/llvm-test/install" \
-DCMAKE_BUILD_TYPE=Release \
-DLLVM_ENABLE_PROJECTS="clang;lld;lldb;clang-tools-extra;polly" \
-DLLVM_ENABLE_RUNTIMES="compiler-rt;libcxx;libcxxabi;libunwind" \
-DLLVM_TOOLCHAIN_TOOLS="llvm-ar;llvm-ranlib;llvm-objdump;llvm-rc;llvm-cvtres;llvm-nm;llvm-strings;llvm-readobj;llvm-dlltool;llvm-pdbutil;llvm-objcopy;llvm-strip;llvm-cov;llvm-profdata;llvm-addr2line;llvm-symbolizer;llvm-windres;llvm-ml;llvm-readelf;llvm-size" \
-DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON \
-DLLVM_INSTALL_BINUTILS_SYMLINKS=OFF \
-DLLVM_PARALLEL_LINK_JOBS=1 \
-DLLVM_TARGETS_TO_BUILD=ARM \
-DLLVM_ENABLE_WARNINGS=OFF \
-DRUNTIMES_armv7l-unknown-linux-gnueabihf_COMPILER_RT_BUILD_GWP_ASAN=OFF \
2>&1 | tee ${HOME}/Work/llvm-test/cmake-output-$(date -u +%Y%m%d-%H%M%S).txt
cmake --build "${HOME}/Work/llvm-test/build" --verbose 2>&1 | tee ${HOME}/Work/llvm-test/build-output-$(date -u +%Y%m%d-%H%M%S).txtIt'll take about 8 hours to know the result. A small question: is the more generic |
|
For completeness, I confirm that a build with the master branch and the same initial configuration, failed with the same error. |
|
We are getting closer, but there is still something wrong with the triple: [1434/1442] Linking CXX static library /home/ilg/Work/llvm-test/build/lib/clang/15.0.7/lib/armhf-unknown-linux-gnueabihf/libclang_rt.fuzzer_interceptors.a
FAILED: /home/ilg/Work/llvm-test/build/lib/clang/15.0.7/lib/armhf-unknown-linux-gnueabihf/libclang_rt.fuzzer_interceptors.a
: && /usr/bin/cmake -E rm -f /home/ilg/Work/llvm-test/build/lib/clang/15.0.7/lib/armhf-unknown-linux-gnueabihf/libclang_rt.fuzzer_interceptors.a && /home/ilg/Work/llvm-test/build/bin/llvm-ar Dqc /home/ilg/Work/llvm-test/build/lib/clang/15.0.7/lib/armhf-unknown-linux-gnueabihf/libclang_rt.fuzzer_interceptors.a compiler-rt/lib/fuzzer/CMakeFiles/RTfuzzer_interceptors.armhf.dir/FuzzerInterceptors.cpp.o && /home/ilg/Work/llvm-test/build/bin/llvm-ranlib -D /home/ilg/Work/llvm-test/build/lib/clang/15.0.7/lib/armhf-unknown-linux-gnueabihf/libclang_rt.fuzzer_interceptors.a && cd /home/ilg/Work/llvm-test/build/runtimes/runtimes-bins/compiler-rt/lib/fuzzer/cxx_armhf_merge.dir && /home/ilg/Work/llvm-test/build/./bin/clang++ --target=armhf-unknown-linux-gnueabihf -march=armv7-a -mfloat-abi=hard -Wl,--whole-archive /home/ilg/Work/llvm-test/build/lib/clang/15.0.7/lib/armhf-unknown-linux-gnueabihf/libclang_rt.fuzzer_interceptors.a -Wl,--no-whole-archive /home/ilg/Work/llvm-test/build/runtimes/runtimes-bins/compiler-rt/lib/fuzzer/libcxx_fuzzer_armhf/lib/libc++.a -r -o fuzzer_interceptors.o && /home/ilg/Work/llvm-test/build/bin/llvm-objcopy --localize-hidden fuzzer_interceptors.o && /usr/bin/cmake -E remove /home/ilg/Work/llvm-test/build/lib/clang/15.0.7/lib/armhf-unknown-linux-gnueabihf/libclang_rt.fuzzer_interceptors.a && /home/ilg/Work/llvm-test/build/bin/llvm-ar qcs /home/ilg/Work/llvm-test/build/lib/clang/15.0.7/lib/armhf-unknown-linux-gnueabihf/libclang_rt.fuzzer_interceptors.a fuzzer_interceptors.o
clang-15: error: unknown target triple 'armhf-unknown-linux-gnueabihf', please use -triple or -arch
[1435/1442] Building CXX object compiler-rt/lib/fuzzer/CMakeFiles/RTfuzzer.armhf.dir/FuzzerUtilLinux.cpp.o
[1436/1442] Building CXX object compiler-rt/lib/fuzzer/CMakeFiles/RTfuzzer_main.armhf.dir/FuzzerMain.cpp.o
[1437/1442] Building CXX object compiler-rt/lib/fuzzer/CMakeFiles/RTfuzzer.armhf.dir/FuzzerUtil.cpp.o
[1438/1442] Building CXX object compiler-rt/lib/fuzzer/CMakeFiles/RTfuzzer.armhf.dir/FuzzerTracePC.cpp.o
[1439/1442] Building CXX object compiler-rt/lib/fuzzer/CMakeFiles/RTfuzzer.armhf.dir/FuzzerUtilPosix.cpp.o
ninja: build stopped: subcommand failed.
[5299/5302] /usr/bin/cmake -E cmake_symlink_library lib/liblldbIntelFeatures.so.15 lib/liblldbIntelFeatures.so.15 lib/liblldbIntelFeatures.so && :
FAILED: runtimes/runtimes-stamps/runtimes-build
cd /home/ilg/Work/llvm-test/build/runtimes/runtimes-bins && /usr/bin/cmake --build .
ninja: build stopped: subcommand failed.
ilg@xbbla32:~ $ /home/ilg/Work/llvm-test/build/./bin/clang++ -v
clang version 15.0.7
Target: armv7l-unknown-linux-gnueabihf
Thread model: posix
InstalledDir: /home/ilg/Work/llvm-test/build/./bin
Found candidate GCC installation: /usr/lib/gcc/arm-linux-gnueabihf/10
Found candidate GCC installation: /usr/lib/gcc/arm-linux-gnueabihf/7
Found candidate GCC installation: /usr/lib/gcc/arm-linux-gnueabihf/7.5.0
Found candidate GCC installation: /usr/lib/gcc/arm-linux-gnueabihf/8
Selected GCC installation: /usr/lib/gcc/arm-linux-gnueabihf/10
Candidate multilib: .;@m32
Selected multilib: .;@m32
ilg@xbbla32:~ $ The machine is a Raspberry Pi OS 32, and I guess clang reports the triple correctly, but CMake used |
My understanding is that because the runtimes build can build multiple targets, options have to include the target name in them so that they are limited to just that target. Options like
I think this is due to The default is currently to disable this for Arm Linux precisely due to it producing an invalid triple like the one you've seen. See https://github.com/llvm/llvm-project/blob/main/llvm/CMakeLists.txt#L820, values from the command line override it (an error here would have been good but it slipped my mind at the time). I do have https://reviews.llvm.org/D140011 to support per target runtime dirs but I am waiting until the 16 branch has happened before landing it because it has had limited testing so far. |
Ok, makes sense. I am optimistic that with this change my build will be fine. Thus I started a production build with the final script that passed on all other platforms (including Linux aarch64); we'll know the result in about 8 hours. |
Great! FYI, I'm using Did you consider a multilib configuration for aarch64 which includes the 32-bit libraries? I know that the compiler can be configured with both architectures, but I could not configure the build to generate the 32-bit libraries. Perhaps your new contributions can fix this too. |
|
Thanks for the diligent work on this @luporl ! |
|
Thanks for all your support @DavidSpickett ! |
|
thanks for the fix, can this please be backported into the 16.x branch? |
EugeneZelenko
added
compiler-rt:scudo
|
@luporl Please cherry-pick this to llvm 16. You can use the cherry-pick command from here to do that, see https://llvm.org/docs/HowToReleaseLLVM.html#triaging-bug-reports-for-releases. If it doesn't apply cleanly you can make a branch and resolve it that way. |
|
@DavidSpickett I cherry-picked it to llvm 16 locally; it applied cleanly, but I'm getting build errors: I suspect we also need to pick your other patch, from https://reviews.llvm.org/D140011, right? |
|
We shouldn't need to, the default on llvm 16 for arm Linux should be ENABLE_PER_TARGET_RUNTIME_DIR=OFF. Try wiping your ccache and rebuilding to double check. I am hesitant to cherry pick that patch if we can avoid it, because it changes a default. If it's the only way to fix things, we could make a version that doesn't change the default instead. |
|
Even with ccache wiped and a clean build, the error persists. We don't need all of D140011, that depends on D139536. Only the change that improves arm arch handling, in CompilerRTUtils.cmake, is needed. I have it in this abandoned patch: https://reviews.llvm.org/D142906. |
|
Yes let's do that. |
The build of scudo was failing on armv7l, with undefined references to unwinder symbols, such as __aeabi_unwind_cpp_pr0. These are needed by RTGwpAsan and thus, on ARM, scudo must also be linked against an unwind library. The cmake command that caused the build failure was: cmake --fresh -S "$PWD/llvm/" -B "$PWD/build/" -G Ninja \ -DCMAKE_INSTALL_PREFIX="$PWD/install" \ -DCMAKE_BUILD_TYPE=Release \ -DLLVM_ENABLE_PROJECTS="clang;lld;lldb;clang-tools-extra;polly" \ -DLLVM_ENABLE_RUNTIMES="compiler-rt;libcxx;libcxxabi;libunwind" \ -DLLVM_TOOLCHAIN_TOOLS="llvm-ar;llvm-ranlib;llvm-objdump;\ llvm-rc;llvm-cvtres;llvm-nm;llvm-strings;llvm-readobj;\ llvm-dlltool;llvm-pdbutil;llvm-objcopy;llvm-strip;llvm-cov;\ llvm-profdata;llvm-addr2line;llvm-symbolizer;llvm-windres;llvm-ml;\ llvm-readelf;llvm-size" \ -DLLVM_INSTALL_BINUTILS_SYMLINKS=OFF -DLLVM_PARALLEL_LINK_JOBS=1 Fixes llvm#60115 Reviewed By: hctim Differential Revision: https://reviews.llvm.org/D142888
The build of scudo was failing on armv7l, with undefined references to unwinder symbols, such as __aeabi_unwind_cpp_pr0. These are needed by RTGwpAsan and thus, on ARM, scudo must also be linked against an unwind library. The cmake command that caused the build failure was: cmake --fresh -S "$PWD/llvm/" -B "$PWD/build/" -G Ninja \ -DCMAKE_INSTALL_PREFIX="$PWD/install" \ -DCMAKE_BUILD_TYPE=Release \ -DLLVM_ENABLE_PROJECTS="clang;lld;lldb;clang-tools-extra;polly" \ -DLLVM_ENABLE_RUNTIMES="compiler-rt;libcxx;libcxxabi;libunwind" \ -DLLVM_TOOLCHAIN_TOOLS="llvm-ar;llvm-ranlib;llvm-objdump;\ llvm-rc;llvm-cvtres;llvm-nm;llvm-strings;llvm-readobj;\ llvm-dlltool;llvm-pdbutil;llvm-objcopy;llvm-strip;llvm-cov;\ llvm-profdata;llvm-addr2line;llvm-symbolizer;llvm-windres;llvm-ml;\ llvm-readelf;llvm-size" \ -DLLVM_INSTALL_BINUTILS_SYMLINKS=OFF -DLLVM_PARALLEL_LINK_JOBS=1 Fixes llvm#60115 Reviewed By: hctim Differential Revision: https://reviews.llvm.org/D142888 (cherry picked from commit e1e9726)
The fuzzer build was failing on armv7l, with an invalid triple error. This happened because CMake's get_compiler_rt_target function was missing some code to correctly handle arm archs, such as armhf. This was originaly part of https://reviews.llvm.org/D140011, that landed on main with commit cd173cb. Fixes llvm#60115 Differential Revision: https://reviews.llvm.org/D142906
|
Could you clarify if all required patches were merged into 16.0.6? |
|
Great! I confirm that my build scripts passed on x86_64/aarch64/armv7l with 16.0.6. Thank you for fixing this. |
The build of scudo was failing on armv7l, with undefined references to unwinder symbols, such as __aeabi_unwind_cpp_pr0. These are needed by RTGwpAsan and thus, on ARM, scudo must also be linked against an unwind library. The cmake command that caused the build failure was: cmake --fresh -S "$PWD/llvm/" -B "$PWD/build/" -G Ninja \ -DCMAKE_INSTALL_PREFIX="$PWD/install" \ -DCMAKE_BUILD_TYPE=Release \ -DLLVM_ENABLE_PROJECTS="clang;lld;lldb;clang-tools-extra;polly" \ -DLLVM_ENABLE_RUNTIMES="compiler-rt;libcxx;libcxxabi;libunwind" \ -DLLVM_TOOLCHAIN_TOOLS="llvm-ar;llvm-ranlib;llvm-objdump;\ llvm-rc;llvm-cvtres;llvm-nm;llvm-strings;llvm-readobj;\ llvm-dlltool;llvm-pdbutil;llvm-objcopy;llvm-strip;llvm-cov;\ llvm-profdata;llvm-addr2line;llvm-symbolizer;llvm-windres;llvm-ml;\ llvm-readelf;llvm-size" \ -DLLVM_INSTALL_BINUTILS_SYMLINKS=OFF -DLLVM_PARALLEL_LINK_JOBS=1 Fixes llvm#60115 Reviewed By: hctim Differential Revision: https://reviews.llvm.org/D142888
I'm maintaining a multi-platform binary distribution (https://github.com/xpack-dev-tools/clang-xpack/) and when I tried to build the latest release I noticed the 32-bit arm build failed (and it was not due to low memory, as feared...).
To exclude issues with my configuration and build environment, I did a separate build with a simple configuration on a clean machine and the build failed too.
The commands used were:
The build failed while making the runtime, due to the missing unwinder library:
The machine used was a Raspberry Pi OS 32-bit.
I kept the entire build environment, and I also have a build with the same configuration on an aarch64 machine (Raspberry Pi OS 64-bit) which passed, so we can use it as a reference.
Any similar experience? I would appreciate any suggestions how to fix this.
The text was updated successfully, but these errors were encountered: