Bazel 4.2.2: C fails to link on macOS x86_64 #14395

uhthomas · 2021-12-08T21:55:54Z

Description of the problem / feature request:

C fails to link on macOS x86_64. This works fine on Bazel 4.2.1 and so I would consider this a regression.

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

I have a repository which contains a reproduction for the issue.

https://github.com/uhthomas/bazel-issue-14395

git clone https://github.com/uhthomas/bazel-issue-14395
cd bazel-issue-14395
USE_BAZEL_VERSION=4.2.2 bazel build //:bin

The project builds okay with Bazel 4.2.1.

USE_BAZEL_VERSION=4.2.1 bazel build //:bin

What operating system are you running Bazel on?

❯ uname -mrs
Darwin 20.6.0 x86_64

What's the output of `bazel info release`?

❯ USE_BAZEL_VERSION=4.2.2 bazel info release
release 4.2.2

Have you found anything relevant by searching the web?

No.

Any other information, logs, or outputs that you want to share?

❯ USE_BAZEL_VERSION=4.2.2 bazel build //:bin
INFO: Analyzed target //:bin (46 packages loaded, 8076 targets configured).
INFO: Found 1 target...
ERROR: /private/var/tmp/_bazel_thomas/9c861fc6d98a2c5a9cf5eae5dc642b60/external/raze__memchr__2_4_1/BUILD.bazel:40:19: Compiling Rust bin memchr_build_script_ v2.4.1 (35 files) failed: (Exit 1): process_wrapper failed: error executing command bazel-out/darwin-opt-exec-2B5CBBC6/bin/external/rules_rust/util/process_wrapper/process_wrapper --subst 'pwd=${pwd}' -- external/rust_darwin_x86_64/bin/rustc external/raze__memchr__2_4_1/build.rs ... (remaining 23 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox
error: linking with `external/local_config_cc/cc_wrapper.sh` failed: exit status: 1
  |
  = note: "external/local_config_cc/cc_wrapper.sh" "-m64" "-arch" "x86_64" "bazel-out/darwin-opt-exec-2B5CBBC6/bin/external/raze__memchr__2_4_1/memchr_build_script_.memchr_build_script_.ae7338a9-cgu.0.rcgu.o" "bazel-out/darwin-opt-exec-2B5CBBC6/bin/external/raze__memchr__2_4_1/memchr_build_script_.memchr_build_script_.ae7338a9-cgu.1.rcgu.o" "bazel-out/darwin-opt-exec-2B5CBBC6/bin/external/raze__memchr__2_4_1/memchr_build_script_.memchr_build_script_.ae7338a9-cgu.10.rcgu.o" "bazel-out/darwin-opt-exec-2B5CBBC6/bin/external/raze__memchr__2_4_1/memchr_build_script_.memchr_build_script_.ae7338a9-cgu.2.rcgu.o" "bazel-out/darwin-opt-exec-2B5CBBC6/bin/external/raze__memchr__2_4_1/memchr_build_script_.memchr_build_script_.ae7338a9-cgu.3.rcgu.o" "bazel-out/darwin-opt-exec-2B5CBBC6/bin/external/raze__memchr__2_4_1/memchr_build_script_.memchr_build_script_.ae7338a9-cgu.4.rcgu.o" "bazel-out/darwin-opt-exec-2B5CBBC6/bin/external/raze__memchr__2_4_1/memchr_build_script_.memchr_build_script_.ae7338a9-cgu.5.rcgu.o" "bazel-out/darwin-opt-exec-2B5CBBC6/bin/external/raze__memchr__2_4_1/memchr_build_script_.memchr_build_script_.ae7338a9-cgu.6.rcgu.o" "bazel-out/darwin-opt-exec-2B5CBBC6/bin/external/raze__memchr__2_4_1/memchr_build_script_.memchr_build_script_.ae7338a9-cgu.7.rcgu.o" "bazel-out/darwin-opt-exec-2B5CBBC6/bin/external/raze__memchr__2_4_1/memchr_build_script_.memchr_build_script_.ae7338a9-cgu.8.rcgu.o" "bazel-out/darwin-opt-exec-2B5CBBC6/bin/external/raze__memchr__2_4_1/memchr_build_script_.memchr_build_script_.ae7338a9-cgu.9.rcgu.o" "bazel-out/darwin-opt-exec-2B5CBBC6/bin/external/raze__memchr__2_4_1/memchr_build_script_.1s71bpdk8li7d4tg.rcgu.o" "-L" "external/rust_darwin_x86_64/lib/rustlib/x86_64-apple-darwin/lib" "-L" "/private/var/tmp/_bazel_thomas/9c861fc6d98a2c5a9cf5eae5dc642b60/external/rust_darwin_x86_64/lib/rustlib/x86_64-apple-darwin/lib" "/private/var/tmp/_bazel_thomas/9c861fc6d98a2c5a9cf5eae5dc642b60/external/rust_darwin_x86_64/lib/rustlib/x86_64-apple-darwin/lib/libstd-dd8a82589e0cba34.rlib" "/private/var/tmp/_bazel_thomas/9c861fc6d98a2c5a9cf5eae5dc642b60/external/rust_darwin_x86_64/lib/rustlib/x86_64-apple-darwin/lib/libpanic_unwind-8c04c8bd0d1a8900.rlib" "/private/var/tmp/_bazel_thomas/9c861fc6d98a2c5a9cf5eae5dc642b60/external/rust_darwin_x86_64/lib/rustlib/x86_64-apple-darwin/lib/libobject-c6a4ae86ed2c40d0.rlib" "/private/var/tmp/_bazel_thomas/9c861fc6d98a2c5a9cf5eae5dc642b60/external/rust_darwin_x86_64/lib/rustlib/x86_64-apple-darwin/lib/libmemchr-f9ab4d1b2e38b05e.rlib" "/private/var/tmp/_bazel_thomas/9c861fc6d98a2c5a9cf5eae5dc642b60/external/rust_darwin_x86_64/lib/rustlib/x86_64-apple-darwin/lib/libaddr2line-002c7b677ad6c512.rlib" "/private/var/tmp/_bazel_thomas/9c861fc6d98a2c5a9cf5eae5dc642b60/external/rust_darwin_x86_64/lib/rustlib/x86_64-apple-darwin/lib/libgimli-a3f3d9f86c37973f.rlib" "/private/var/tmp/_bazel_thomas/9c861fc6d98a2c5a9cf5eae5dc642b60/external/rust_darwin_x86_64/lib/rustlib/x86_64-apple-darwin/lib/libstd_detect-8b14bcf2354140fd.rlib" "/private/var/tmp/_bazel_thomas/9c861fc6d98a2c5a9cf5eae5dc642b60/external/rust_darwin_x86_64/lib/rustlib/x86_64-apple-darwin/lib/librustc_demangle-d6f2fd91ec8bbbcc.rlib" "/private/var/tmp/_bazel_thomas/9c861fc6d98a2c5a9cf5eae5dc642b60/external/rust_darwin_x86_64/lib/rustlib/x86_64-apple-darwin/lib/libhashbrown-24c80e37fb5b15c5.rlib" "/private/var/tmp/_bazel_thomas/9c861fc6d98a2c5a9cf5eae5dc642b60/external/rust_darwin_x86_64/lib/rustlib/x86_64-apple-darwin/lib/librustc_std_workspace_alloc-edb9b11fa36b4795.rlib" "/private/var/tmp/_bazel_thomas/9c861fc6d98a2c5a9cf5eae5dc642b60/external/rust_darwin_x86_64/lib/rustlib/x86_64-apple-darwin/lib/libunwind-769780536fb7ef9b.rlib" "/private/var/tmp/_bazel_thomas/9c861fc6d98a2c5a9cf5eae5dc642b60/external/rust_darwin_x86_64/lib/rustlib/x86_64-apple-darwin/lib/libcfg_if-d37c37a3a3ac2b0c.rlib" "/private/var/tmp/_bazel_thomas/9c861fc6d98a2c5a9cf5eae5dc642b60/external/rust_darwin_x86_64/lib/rustlib/x86_64-apple-darwin/lib/liblibc-c1bdc4c1f89760ef.rlib" "/private/var/tmp/_bazel_thomas/9c861fc6d98a2c5a9cf5eae5dc642b60/external/rust_darwin_x86_64/lib/rustlib/x86_64-apple-darwin/lib/liballoc-750380e9c94de9ce.rlib" "/private/var/tmp/_bazel_thomas/9c861fc6d98a2c5a9cf5eae5dc642b60/external/rust_darwin_x86_64/lib/rustlib/x86_64-apple-darwin/lib/librustc_std_workspace_core-1108e622f5a15c3d.rlib" "/private/var/tmp/_bazel_thomas/9c861fc6d98a2c5a9cf5eae5dc642b60/external/rust_darwin_x86_64/lib/rustlib/x86_64-apple-darwin/lib/libcore-43af7053e70b1eed.rlib" "/private/var/tmp/_bazel_thomas/9c861fc6d98a2c5a9cf5eae5dc642b60/external/rust_darwin_x86_64/lib/rustlib/x86_64-apple-darwin/lib/libcompiler_builtins-3a81ebf6a3abbdee.rlib" "-lSystem" "-lresolv" "-lc" "-lm" "-liconv" "-L" "/private/var/tmp/_bazel_thomas/9c861fc6d98a2c5a9cf5eae5dc642b60/external/rust_darwin_x86_64/lib/rustlib/x86_64-apple-darwin/lib" "-o" "bazel-out/darwin-opt-exec-2B5CBBC6/bin/external/raze__memchr__2_4_1/memchr_build_script_" "-Wl,-dead_strip" "-nodefaultlibs" "-undefined" "dynamic_lookup" "-headerpad_max_install_names" "-lstdc++" "-lm"
  = note: ld: library not found for -lstdc++
          clang: error: linker command failed with exit code 1 (use -v to see invocation)


error: aborting due to previous error

Target //:bin failed to build
Use --verbose_failures to see the command lines of failed build steps.
ERROR: /private/var/tmp/_bazel_thomas/9c861fc6d98a2c5a9cf5eae5dc642b60/external/raze__memchr__2_4_1/BUILD.bazel:65:13 Compiling Rust rlib memchr v2.4.1 (35 files) failed: (Exit 1): process_wrapper failed: error executing command bazel-out/darwin-opt-exec-2B5CBBC6/bin/external/rules_rust/util/process_wrapper/process_wrapper --subst 'pwd=${pwd}' -- external/rust_darwin_x86_64/bin/rustc external/raze__memchr__2_4_1/build.rs ... (remaining 23 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox
INFO: Elapsed time: 12.594s, Critical Path: 2.18s
INFO: 46 processes: 40 internal, 6 darwin-sandbox.
FAILED: Build did NOT complete successfully

The text was updated successfully, but these errors were encountered:

gabrielrussoc · 2021-12-09T10:55:07Z

maybe bazelbuild/rules_rust#978?

4.2.2 includes #14319, which says "This will cause the values passed in the link_libs parameters to not get silently dropped on macOS 12 Monterey" - so I wonder if this just surfaced a problem in rules_rust

uhthomas · 2021-12-09T11:07:18Z

Thanks for taking a look! I had assumed #14319 may have been related, but not sure how or in what capacity.

"This will cause the values passed in the link_libs parameters to not get silently dropped on macOS 12 Monterey"

I'm on Big Sur.

aiuto · 2021-12-15T04:47:01Z

Do you have a reproduction that does not involve rust?

Silic0nS0ldier · 2022-01-08T07:10:54Z

Similar problem has been reported elsewhere.

Consensus seems to be that -lc++ is what should be used on MacOS.
EDIT: libstdc++ deprecated in XCode 8, removed in XCode 10 (see XCode 10 release notes).

I'm not sure where the flags come from, but I know they are put into the failing command at:
https://github.com/bazelbuild/rules_rust/blob/412826645587d2c5ea8e30a98522c25a8a53ae0c/rust/private/rustc.bzl#L648

As an experiment I tried replacing the arg with the apparent new version, and the issue was resolved.

fixed_link_args = [x for x in link_args if x != "-lstdc++"]
fixed_link_args.append("-lc++")
rustc_flags.add_joined("--codegen", fixed_link_args, join_with = " ", format_joined = "link-args=%s")

Removing the line altogether also worked (simple program though, so lib may just not have been needed).

Silic0nS0ldier · 2022-01-08T14:28:46Z

Did some more digging and found where -lstdc++ was coming from. An external repository generated by cc_autoconf, called local_config_cc in my case.

Rule definition is at

bazel/tools/cpp/cc_configure.bzl

Lines 145 to 179 in b4b0c32

    
           cc_autoconf = repository_rule( 
        
               environ = [ 
        
                   "ABI_LIBC_VERSION", 
        
                   "ABI_VERSION", 
        
                   "BAZEL_COMPILER", 
        
                   "BAZEL_HOST_SYSTEM", 
        
                   "BAZEL_CXXOPTS", 
        
                   "BAZEL_LINKOPTS", 
        
                   "BAZEL_LINKLIBS", 
        
                   "BAZEL_LLVM_COV", 
        
                   "BAZEL_PYTHON", 
        
                   "BAZEL_SH", 
        
                   "BAZEL_TARGET_CPU", 
        
                   "BAZEL_TARGET_LIBC", 
        
                   "BAZEL_TARGET_SYSTEM", 
        
                   "BAZEL_USE_CPP_ONLY_TOOLCHAIN", 
        
                   "BAZEL_USE_XCODE_TOOLCHAIN", 
        
                   "BAZEL_DO_NOT_DETECT_CPP_TOOLCHAIN", 
        
                   "BAZEL_USE_LLVM_NATIVE_COVERAGE", 
        
                   "BAZEL_LLVM", 
        
                   "BAZEL_IGNORE_SYSTEM_HEADERS_VERSIONS", 
        
                   "USE_CLANG_CL", 
        
                   "CC", 
        
                   "CC_CONFIGURE_DEBUG", 
        
                   "CC_TOOLCHAIN_NAME", 
        
                   "CPLUS_INCLUDE_PATH", 
        
                   "DEVELOPER_DIR", 
        
                   "GCOV", 
        
                   "HOMEBREW_RUBY_PATH", 
        
                   "SYSTEMROOT", 
        
                   "USER", 
        
               ] + MSVC_ENVVARS, 
        
               implementation = cc_autoconf_impl, 
        
               configure = True, 
        
           )

With the more limited search range I was able to identify 2 likely places -lstdc++ was coming from.

bazel/tools/cpp/cc_toolchain_config.bzl

Line 398 in c579155

"-lstdc++",

bazel/tools/cpp/unix_cc_configure.bzl

Line 403 in c579155

bazel_linkopts = "-lstdc++:-lm"

Argument ordering has me thinking bazel/tools/cpp/cc_toolchain_config.bzl is the ultimate source (especially since configure_osx_toolchain calls out to it). If true, then darwin may need to be handled differently.

keith · 2022-01-11T00:44:31Z

Note for macOS users with Xcode installed, you have to set BAZEL_USE_CPP_ONLY_TOOLCHAIN=1 in the environment to trigger this repro case.

I looked into this a bit in clang. So what seems to be the issue is that -lstdc++ is just invalid on darwin, but clang knows how to remap it to -lc++ for the linker. For example:

% clang -lstdc++ -###
Apple clang version 13.0.0 (clang-1300.0.29.3)
Target: x86_64-apple-darwin21.2.0
 "/Applications/Xcode-13.0.0-RC1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld" "-demangle" "-lto_library" "/Applications/Xcode-13.0.0-RC1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/libLTO.dylib" "-dynamic" "-arch" "x86_64" "-platform_version" "macos" "12.0.0" "12.1" "-syslibroot" "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk" "-o" "a.out" "-L/usr/local/lib" "-lc++" "-lSystem" "/Applications/Xcode-13.0.0-RC1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/13.0.0/lib/darwin/libclang_rt.osx.a"

You can see the arg passed to the linker here is different from what I passed. The issue in this case -nodefaultlibs is also passed, which changes this behavior and doesn't remap that argument (trace the logic for this starting here):

% clang -lstdc++ -### -nodefaultlibs
Apple clang version 13.0.0 (clang-1300.0.29.3)
InstalledDir: /Applications/Xcode-13.0.0-RC1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
 "/Applications/Xcode-13.0.0-RC1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld" "-demangle" "-lto_library" "/Applications/Xcode-13.0.0-RC1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/libLTO.dylib" "-dynamic" "-arch" "x86_64" "-platform_version" "macos" "12.0.0" "12.1" "-syslibroot" "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk" "-o" "a.out" "-L/usr/local/lib" "-lstdc++"

Based on this comment https://github.com/llvm/llvm-project/blob/11a46b174923a2509ac620bc8ff621ecdf6135b6/clang/lib/Driver/ToolChains/Darwin.cpp#L872-L876 this has been invalid for a long time.

It looks like -nodefaultlibs is coming from rust itself https://github.com/rust-lang/rust/blob/3d57c61a9e04dcd3df633f41142009d6dcad4399/compiler/rustc_codegen_ssa/src/back/linker.rs#L635-L637 and raze is pulling the flags it passes there. This in itself seems fine, but explains why this bug surfaces with rust builds specifically.

Here's a patch to fix this that I think is safe: #14542

Can folks try the bazel built with my branch to verify? It does fix the repro case above.

This flag has been invalid seemingly since macOS 10.9 from 2013. By default clang remaps this back to -lc++, but if your linkopts pass -nodefaultlibs this is not the case, in which case this invalid flag fails the link. This happened for rust builds where the rust driver passes this flag intentionally. For more info see bazelbuild#14395 (comment)

Silic0nS0ldier · 2022-01-11T11:49:29Z

With BAZEL_USE_CPP_ONLY_TOOLCHAIN=1 in place to bypass my newly minted XCode install (to compile Bazel from your PR branch), I was able to confirm this fixes the issue for both @uhthomas's reproduction repo and my instance of the issue.

Thanks @keith!

This flag has been invalid seemingly since macOS 10.9 from 2013. By default clang remaps this back to -lc++, but if your linkopts pass -nodefaultlibs this is not the case, in which case this invalid flag fails the link. This happened for rust builds where the rust driver passes this flag intentionally. For more info see #14395 (comment) Fixes google/cargo-raze#247 #14395 bazelbuild/rules_rust#226 Closes #14542. PiperOrigin-RevId: 424588734

This flag has been invalid seemingly since macOS 10.9 from 2013. By default clang remaps this back to -lc++, but if your linkopts pass -nodefaultlibs this is not the case, in which case this invalid flag fails the link. This happened for rust builds where the rust driver passes this flag intentionally. For more info see bazelbuild#14395 (comment) Fixes google/cargo-raze#247 bazelbuild#14395 bazelbuild/rules_rust#226 Closes bazelbuild#14542. PiperOrigin-RevId: 424588734 (cherry picked from commit 8b60c90)

This flag has been invalid seemingly since macOS 10.9 from 2013. By default clang remaps this back to -lc++, but if your linkopts pass -nodefaultlibs this is not the case, in which case this invalid flag fails the link. This happened for rust builds where the rust driver passes this flag intentionally. For more info see #14395 (comment) Fixes google/cargo-raze#247 #14395 bazelbuild/rules_rust#226 Closes #14542. PiperOrigin-RevId: 424588734 (cherry picked from commit 8b60c90) Co-authored-by: Keith Smiley <keithbsmiley@gmail.com>

keith · 2022-04-02T00:18:33Z

We can close this one now as the fixes have been merged into bazel 5.1

aiuto assigned uhthomas Dec 15, 2021

aiuto added more data needed team-Rules-CPP Issues for C++ rules labels Dec 15, 2021

Silic0nS0ldier mentioned this issue Jan 9, 2022

libstdc++ changes Silic0nS0ldier/bazel#1

Closed

This was referenced Jan 11, 2022

Remove uses of -lstdc++ on darwin #14542

Closed

macos: ld: library not found for -lstdc++ when compiling no_deps example google/cargo-raze#247

Open

bazel build //... fails on OSX bazelbuild/rules_rust#226

Closed

gregestren added the type: bug label Jan 31, 2022

brentleyjones mentioned this issue Feb 8, 2022

[5.1] Remove uses of -lstdc++ on darwin #14750

Merged

pawelchcki mentioned this issue Feb 18, 2022

Experiments with bazel build system DataDog/libddprof#24

Draft

6 tasks

kobigurk mentioned this issue Feb 25, 2022

Error when building rust examples on macos m1 risc0/risc0#4

Closed

uhthomas closed this as completed Apr 19, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bazel 4.2.2: C fails to link on macOS x86_64 #14395

Bazel 4.2.2: C fails to link on macOS x86_64 #14395

uhthomas commented Dec 8, 2021 •

edited

Loading

gabrielrussoc commented Dec 9, 2021 •

edited

Loading

uhthomas commented Dec 9, 2021

aiuto commented Dec 15, 2021

Silic0nS0ldier commented Jan 8, 2022 •

edited

Loading

Silic0nS0ldier commented Jan 8, 2022

keith commented Jan 11, 2022 •

edited

Loading

Silic0nS0ldier commented Jan 11, 2022

keith commented Apr 2, 2022

Bazel 4.2.2: C fails to link on macOS x86_64 #14395

Bazel 4.2.2: C fails to link on macOS x86_64 #14395

Comments

uhthomas commented Dec 8, 2021 • edited Loading

Description of the problem / feature request:

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

What operating system are you running Bazel on?

What's the output of bazel info release?

Have you found anything relevant by searching the web?

Any other information, logs, or outputs that you want to share?

gabrielrussoc commented Dec 9, 2021 • edited Loading

uhthomas commented Dec 9, 2021

aiuto commented Dec 15, 2021

Silic0nS0ldier commented Jan 8, 2022 • edited Loading

Silic0nS0ldier commented Jan 8, 2022

keith commented Jan 11, 2022 • edited Loading

Silic0nS0ldier commented Jan 11, 2022

keith commented Apr 2, 2022

uhthomas commented Dec 8, 2021 •

edited

Loading

What's the output of `bazel info release`?

gabrielrussoc commented Dec 9, 2021 •

edited

Loading

Silic0nS0ldier commented Jan 8, 2022 •

edited

Loading

keith commented Jan 11, 2022 •

edited

Loading