CoreCLR native build on Linux / Linux_musl arm64 / arm causes clang crashes #84802
Comments
trylek
added
area-Infrastructure-coreclr
blocking-outerloop
ghost
added
the
|
Tagging subscribers to this area: @hoyosjs Issue DetailsOS: Linux / Linux_musl Going over the recent runs it seems to be pretty recent - first repro I've found is from yesterday afternoon: Diagnostics: clang crashes when compiling PLEASE submit a bug report to https://bugs.llvm.org/ and include the crash backtrace, preprocessed source, and associated run script. Stack dump: 0. Program arguments: /usr/local/bin/clang-12 --target=arm-linux-gnueabihf --gcc-toolchain=/crossrootfs/arm/usr --sysroot=/crossrootfs/arm -DBUILDENV_DEBUG=1 -DCLR_ARM_FPU_CAPABILITY=0x7 -DDEBUG -DDISABLE_CONTRACTS -DHAVE_CONFIG_H=1 -DHAVE___THREAD=0 -DHOST_ARM -DHOST_UNIX -DPACKAGE_BUGREPORT=\"\" -DPACKAGE_STRING=\"\" -DTARGET_32BIT -DTARGET_ARM -DTARGET_LINUX -DTARGET_UNIX -DURTBLDENV_FRIENDLY=Debug -DUSE_STL -D_DBG -D_DEBUG -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -Darm_search_unwind_table=UNW_OBJ(arm_search_unwind_table) -Dasm=__asm__ -I/__w/1/s/artifacts/obj/external/libunwind -I/__w/1/s/src/native/external/libunwind_extras -I/__w/1/s/src/native -I/__w/1/s/src/coreclr/pal/inc -I/__w/1/s/src/coreclr/pal/src -I/__w/1/s/src/coreclr/pal/../inc -I/__w/1/s/src/native/external/libunwind/include -I/__w/1/s/src/native/external/libunwind/include/tdep -I/__w/1/s/artifacts/obj/external/libunwind/include -I/__w/1/s/artifacts/obj/external/libunwind/include/tdep -I/__w/1/s/src/native/external/libunwind/include/tdep-arm -I/__w/1/s/src/native/external/libunwind/src -g -fPIC -mthumb -mfpu=vfpv3 -O0 -g -Wall -Wno-null-conversion -fno-omit-frame-pointer -fms-extensions -fwrapv -fstack-protector-strong -Werror -Wno-unused-variable -Wno-unused-value -Wno-unused-function -Wno-tautological-compare -Wno-unknown-pragmas -Wimplicit-fallthrough -Wno-unused-but-set-variable -ffp-contract=off -Wno-unknown-warning-option -ferror-limit=4096 -Wno-unused-private-field -Wno-constant-logical-operand -Wno-pragma-pack -Wno-incompatible-ms-struct -Wno-reserved-identifier -Wno-unsafe-buffer-usage -Wno-single-bit-bitfield-constant-conversion -Wno-cast-function-type-strict -fsigned-char -fvisibility=hidden -ffunction-sections -march=armv7-a -fexceptions -Wno-format -Wno-format-security -Wno-implicit-fallthrough -Wno-header-guard -Wno-absolute-value -Wno-inline-asm -Wno-implicit-function-declaration -Wno-implicit-int-conversion -std=gnu11 -MD -MT /__w/1/s/artifacts/obj/external/libunwind/CMakeFiles/libunwind.dir/__w/1/s/src/native/external/libunwind/src/elf32.c.o -MF CMakeFiles/libunwind.dir/__w/1/s/src/native/external/libunwind/src/elf32.c.o.d -o CMakeFiles/libunwind.dir/__w/1/s/src/native/external/libunwind/src/elf32.c.o -c /__w/1/s/src/native/external/libunwind/src/elf32.c 1. parser at end of file 2. Code generation [ 8%] Building CXX object md/enc/CMakeFiles/mdruntimerw_wks_obj.dir/cmake_pch.hxx.pch #0 0x000056228ba9805e llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/usr/local/bin/clang-12+0x213f05e) #1 0x000056228ba95e54 llvm::sys::RunSignalHandlers() (/usr/local/bin/clang-12+0x213ce54) #2 0x000056228ba0c438 CrashRecoverySignalHandler(int) (/usr/local/bin/clang-12+0x20b3438) #3 0x00007f29380e9e30 __restore_rt (/lib/libc.so.6+0x42e30) #4 0x000056228c53e310 llvm::DIE::getUnitDie() const (/usr/local/bin/clang-12+0x2be5310) #5 0x000056228c54869c llvm::DwarfDebug::finishEntityDefinitions() (/usr/local/bin/clang-12+0x2bef69c) #6 0x000056228c561a52 llvm::DwarfDebug::finalizeModuleInfo() (/usr/local/bin/clang-12+0x2c08a52) #7 0x000056228c564a68 llvm::DwarfDebug::endModule() (/usr/local/bin/clang-12+0x2c0ba68) #8 0x000056228c535aa9 llvm::AsmPrinter::doFinalization(llvm::Module&) (/usr/local/bin/clang-12+0x2bdcaa9) #9 0x000056228b3ef55d llvm::FPPassManager::doFinalization(llvm::Module&) (.localalias) (/usr/local/bin/clang-12+0x1a9655d) #10 0x000056228b3fb020 llvm::legacy::PassManagerImpl::run(llvm::Module&) (/usr/local/bin/clang-12+0x1aa2020) #11 0x000056228bd2bf2c (anonymous namespace)::EmitAssemblyHelper::EmitAssembly(clang::BackendAction, std::unique_ptr >) (.constprop.0) (/usr/local/bin/clang-12+0x23d2f2c) #12 0x000056228bd2dc7c clang::EmitBackendOutput(clang::DiagnosticsEngine&, clang::HeaderSearchOptions const&, clang::CodeGenOptions const&, clang::TargetOptions const&, clang::LangOptions const&, llvm::DataLayout const&, llvm::Module*, clang::BackendAction, std::unique_ptr >) (/usr/local/bin/clang-12+0x23d4c7c) #13 0x000056228c96670c clang::BackendConsumer::HandleTranslationUnit(clang::ASTContext&) (/usr/local/bin/clang-12+0x300d70c) #14 0x000056228d430b39 clang::ParseAST(clang::Sema&, bool, bool) (/usr/local/bin/clang-12+0x3ad7b39) #15 0x000056228c33b269 clang::FrontendAction::Execute() (/usr/local/bin/clang-12+0x29e2269) #16 0x000056228c2deccb clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) (/usr/local/bin/clang-12+0x2985ccb) #17 0x000056228c3e9c58 clang::ExecuteCompilerInvocation(clang::CompilerInstance*) (/usr/local/bin/clang-12+0x2a90c58) #18 0x000056228a50788f cc1_main(llvm::ArrayRef, char const*, void*) (/usr/local/bin/clang-12+0xbae88f) #19 0x000056228a5052a0 ExecuteCC1Tool(llvm::SmallVectorImpl&) (/usr/local/bin/clang-12+0xbac2a0) #20 0x000056228c18d5b5 void llvm::function_ref::callback_fn >, std::__cxx11::basic_string, std::allocator >*, bool*) const::'lambda'()>(long) (/usr/local/bin/clang-12+0x28345b5) #21 0x000056228ba0c533 llvm::CrashRecoveryContext::RunSafely(llvm::function_ref) (/usr/local/bin/clang-12+0x20b3533) #22 0x000056228c18e6df clang::driver::CC1Command::Execute(llvm::ArrayRef >, std::__cxx11::basic_string, std::allocator >*, bool*) const (.part.0) (/usr/local/bin/clang-12+0x28356df) #23 0x000056228c164ad7 clang::driver::Compilation::ExecuteCommand(clang::driver::Command const&, clang::driver::Command const*&) const (/usr/local/bin/clang-12+0x280bad7) #24 0x000056228c165519 clang::driver::Compilation::ExecuteJobs(clang::driver::JobList const&, llvm::SmallVectorImpl >&) const (/usr/local/bin/clang-12+0x280c519) #25 0x000056228c172f29 clang::driver::Driver::ExecuteCompilation(clang::driver::Compilation&, llvm::SmallVectorImpl >&) (/usr/local/bin/clang-12+0x2819f29) #26 0x000056228a47ecb2 main (/usr/local/bin/clang-12+0xb25cb2) #27 0x00007f29380d557d __libc_start_call_main (/lib/libc.so.6+0x2e57d) #28 0x00007f29380d5630 __libc_start_main_impl (/lib/libc.so.6+0x2e630) #29 0x000056228a504c85 _start (/usr/local/bin/clang-12+0xbabc85) clang-12: error: clang frontend command failed with exit code 139 (use -v to see invocation) clang version 12.0.1 Target: arm-unknown-linux-gnueabihf Thread model: posix InstalledDir: /usr/local/bin [ 8%] Building CXX object md/compiler/CMakeFiles/mdcompiler_wks_obj.dir/assemblymd.cpp.o clang-12: note: diagnostic msg: ******************** PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT: Preprocessed source(s) and associated run script(s) are located at: clang-12: note: diagnostic msg: /tmp/elf32-2e3faa.c clang-12: note: diagnostic msg: /tmp/elf32-2e3faa.sh clang-12: note: diagnostic msg: ******************** I must admit I don't yet fully understand what the "parser at end of file" refers to, whether it's the compiler command line or the source file. /cc @dotnet/runtime-infrastructure @janvorli
|
|
This looks like #84503. I was seeing it in linux arm64 mono build, but hadn't seen it in other builds. dotnet/dotnet-buildtools-prereqs-docker#842 seemed to fix it for me. But the images in dotnet-buildtools-prereqs-docker have already been updated to llvm 16, which we are not yet ready to use. I think we'll want to roll back the images to llvm 12 to fix this ASAP. |
|
@sbomer I am a bit confused - the failure logged above was caused by clang 12, why do we need to revert the upgrade to clang 16? |
|
Runtime is currently building with a set of mariner images that have clang 12, built with gcc. There is no published set of images that has clang 12, built with clang - because the llvm 16 upgrade went in before the build-with-clang fix. So I'm reverting the upgrade temporarily to get us a new set of images that have clang-built clang 12. |
|
Ah, got it, thank you for the explanation. |
|
Frankly speaking I'm not getting why it doesn't fail the same way in the runtime (PR / CI) pipeline, e.g. in this Egor's PR run started about half an hour ago: https://dev.azure.com/dnceng-public/public/_build/results?buildId=238687&view=results |
|
I'm not sure either - I was able to reproduce it consistently when I saw the failure building mono arm64. But it might be very sensitive to the inputs. |
|
I believe this is fixed with #84851. Please reopen if you hit this again. |
OS: Linux / Linux_musl
Architecture: arm64, arm
Example run:
https://dev.azure.com/dnceng-public/public/_build/results?buildId=237422&view=logs&j=fd9dce65-fed6-5f8b-f099-1d407f24e80f&t=73b092a7-fc6e-5522-101b-81c9e52247b8
Going over the recent runs it seems to be pretty recent - first repro I've found is from yesterday afternoon:
https://dev.azure.com/dnceng-public/public/_build/results?buildId=237112&view=results
Diagnostics: clang crashes when compiling
src/native/external/libunwind/src/elf32.corelf64.cbased on targeting architecture. I don't see any recent changes to their source code so I tend to suspect either a clang version update or some scripting update. The problem may not be fully deterministic - the above example run is the one where this also happens on Linux arm, in the remaining runs I saw it was only reproing on Linux / Linux_musl arm64.I must admit I don't yet fully understand what the "parser at end of file" refers to, whether it's the compiler command line or the source file.
/cc @dotnet/runtime-infrastructure @janvorli
The text was updated successfully, but these errors were encountered: