Cannot build cc_test() targets on macOS

[coryan]

Description of the bug:

Starting with Bazel v7 we cannot build cc_test() targets on macOS.

Which category does this issue belong to?

No response

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Use the following repro:

repro.tar.gz

Which operating system are you running Bazel on?

uname -a
Darwin coryan-macbookpro.roam.internal 23.2.0 Darwin Kernel Version 23.2.0: Wed Nov 15 21:53:18 PST 2023; root:xnu-10002.61.3~2/RELEASE_ARM64_T6000 arm64

What is the output of bazel info release?

release 7.0.0

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?

No response

Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.

No response

Have you found anything relevant by searching the web?

There are older bugs with similar problems, e.g. #15069

Any other information, logs, or outputs that you want to share?

xcode-select -p
/Library/Developer/CommandLineTools

c++ --version
Apple clang version 15.0.0 (clang-1500.1.0.2.5)
Target: arm64-apple-darwin23.2.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

[comius]

Can you post the log/error that Bazel gives you?

[coryan]

~/bazelisk build ...
Starting local Bazel server and connecting to it...
INFO: Analyzed 2 targets (43 packages loaded, 241 targets configured).
ERROR: /Users/coryan/repro-pr13274/BUILD.bazel:7:8: Linking test failed: (Exit 1): cc_wrapper.sh failed: error executing CppLink command (from target //:test) external/local_config_cc/cc_wrapper.sh -o bazel-out/darwin_arm64-fastbuild/bin/test -Xlinker -rpath -Xlinker @loader_path/_solib_darwin/ -Xlinker -rpath -Xlinker ... (remaining 8 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
external/local_config_cc/cc_wrapper.sh: line 54: loader_path/_solib_darwin/: No such file or directory
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 3.294s, Critical Path: 0.06s
INFO: 2 processes: 2 internal.
ERROR: Build did NOT complete successfully

The compiler invocation may be of interest:

~/bazelisk build -s ...
INFO: Analyzed 2 targets (0 packages loaded, 0 targets configured).
SUBCOMMAND: # //:test [action 'Linking test', configuration: 7a4f3b2564699f7b8bd4cc135c50d0547ebe04e011e302478e62038755eda25c, execution platform: @@local_config_platform//:host, mnemonic: CppLink]
(cd /private/var/tmp/_bazel_coryan/d7a3664b39ea78b15e7a6b2b3b69cf05/execroot/repro && \
  exec env - \
    PATH='/Users/coryan/Library/Caches/bazelisk/downloads/sha256/b1cf1c5783fa3eac60942bf2ec6169cfccecd2fa5a355587d8d35983bc1e1310/bin:/Users/coryan/Library/Application Support/cloud-code/installer/google-cloud-sdk/bin:/usr/local/git/git-google/bin:/usr/local/git/current/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Users/coryan/Library/Application Support/cloud-code/installer/google-cloud-sdk/bin' \
    PWD=/proc/self/cwd \
  external/local_config_cc/cc_wrapper.sh -o bazel-out/darwin_arm64-fastbuild/bin/test -Xlinker -rpath -Xlinker @loader_path/_solib_darwin/ -Xlinker -rpath -Xlinker @loader_path/test.runfiles/repro/_solib_darwin/ -Lbazel-out/darwin_arm64-fastbuild/bin/_solib_darwin bazel-out/darwin_arm64-fastbuild/bin/_objs/test/test.pic.o -llibgreeting -Wl,-S -undefined dynamic_lookup -headerpad_max_install_names)
# Configuration: 7a4f3b2564699f7b8bd4cc135c50d0547ebe04e011e302478e62038755eda25c
# Execution platform: @@local_config_platform//:host
ERROR: /Users/coryan/repro-pr13274/BUILD.bazel:7:8: Linking test failed: (Exit 1): cc_wrapper.sh failed: error executing CppLink command (from target //:test) external/local_config_cc/cc_wrapper.sh -o bazel-out/darwin_arm64-fastbuild/bin/test -Xlinker -rpath -Xlinker @loader_path/_solib_darwin/ -Xlinker -rpath -Xlinker ... (remaining 8 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
external/local_config_cc/cc_wrapper.sh: line 54: loader_path/_solib_darwin/: No such file or directory
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 0.162s, Critical Path: 0.03s
INFO: 2 processes: 2 internal.
ERROR: Build did NOT complete successfully

Note that the -Xlinker @loader_path/_solib_darwin/ options. The script seems to interpret all options starting with @ as response files:

https://github.com/bazelbuild/rules_cc/blob/51b77439a109b8c8f75ad70a130293291ed8b851/cc/private/toolchain/osx_cc_wrapper.sh.tpl#L54-L63

But on macOS @loader_path has a different meaning.

[keith]

A quick fix here is the rules_cc_toolchains() call in the WORKSPACE needs to move to the bottom, that way the toolchain from apple_support is preferred and things build correctly.

The problem is rules_cc is still registering its own out of date toolchain when done through the WORKSPACE https://github.com/bazelbuild/rules_cc/blob/51b77439a109b8c8f75ad70a130293291ed8b851/cc/repositories.bzl#L9-L10 but with bzlmod it's using the one in bazel https://github.com/bazelbuild/rules_cc/blob/51b77439a109b8c8f75ad70a130293291ed8b851/MODULE.bazel#L9-L10

Likely should just remove the rules_cc_toolchains() call from your WORKSPACE for now

[keith]

this fixes though bazelbuild/rules_cc#215

[coryan]

A quick fix here is the rules_cc_toolchains() call in the WORKSPACE needs to move to the bottom, that way the toolchain from apple_support is preferred and things build correctly.

I tried this change:

diff --git a/WORKSPACE.bazel b/WORKSPACE.bazel
index 681b370..4dac4a4 100644
--- a/WORKSPACE.bazel
+++ b/WORKSPACE.bazel
@@ -35,15 +35,6 @@ maybe(
     sha256 = "34c41bfb59cdaea29ac2df5a2fa79e5add609c71bb303b2ebb10985f93fa20e7",
 )
 
-load(
-    "@rules_cc//cc:repositories.bzl",
-    "rules_cc_dependencies",
-    "rules_cc_toolchains")
-
-rules_cc_dependencies()
-
-rules_cc_toolchains()
-
 load(
     "@build_bazel_rules_apple//apple:repositories.bzl",
     "apple_rules_dependencies",
@@ -58,3 +49,11 @@ load(
 
 apple_support_dependencies()
 
+load(
+    "@rules_cc//cc:repositories.bzl",
+    "rules_cc_dependencies",
+    "rules_cc_toolchains")
+
+rules_cc_dependencies()
+
+rules_cc_toolchains()

And got:

INFO: Starting clean (this may take a while). Consider using --async if the clean takes more than several minutes.
Starting local Bazel server and connecting to it...
ERROR: /private/var/tmp/_bazel_coryan/d7a3664b39ea78b15e7a6b2b3b69cf05/external/local_config_apple_cc/BUILD:79:24: in cc_toolchain_config rule @@local_config_apple_cc//:darwin_arm64: 
Traceback (most recent call last):
        File "/private/var/tmp/_bazel_coryan/d7a3664b39ea78b15e7a6b2b3b69cf05/external/local_config_apple_cc/cc_toolchain_config.bzl", line 2465, column 58, in _impl
                enabled = xcode_support.is_xcode_at_least_version(xcode_config, "15.0.0"),
        File "/private/var/tmp/_bazel_coryan/d7a3664b39ea78b15e7a6b2b3b69cf05/external/build_bazel_apple_support/lib/xcode_support.bzl", line 35, column 13, in _is_xcode_at_least_version
                fail("Could not determine Xcode version at all. This likely means Xcode isn't available; " +
Error in fail: Could not determine Xcode version at all. This likely means Xcode isn't available; if you think this is a mistake, please file an issue.
ERROR: /private/var/tmp/_bazel_coryan/d7a3664b39ea78b15e7a6b2b3b69cf05/external/local_config_apple_cc/BUILD:79:24: Analysis of target '@@local_config_apple_cc//:darwin_arm64' failed
ERROR: Analysis of target '//:greeting' failed; build aborted: Analysis failed
INFO: Elapsed time: 9.857s, Critical Path: 0.02s
INFO: 1 process: 1 internal.
ERROR: Build did NOT complete successfully

I tried this too:

diff --git a/WORKSPACE.bazel b/WORKSPACE.bazel
index 681b370..99723e5 100644
--- a/WORKSPACE.bazel
+++ b/WORKSPACE.bazel
@@ -42,8 +42,6 @@ load(
 
 rules_cc_dependencies()
 
-rules_cc_toolchains()
-
 load(
     "@build_bazel_rules_apple//apple:repositories.bzl",
     "apple_rules_dependencies",
@@ -58,3 +56,4 @@ load(
 
 apple_support_dependencies()
 
+rules_cc_toolchains()

Same results:

INFO: Starting clean (this may take a while). Consider using --async if the clean takes more than several minutes.
Starting local Bazel server and connecting to it...
ERROR: /private/var/tmp/_bazel_coryan/d7a3664b39ea78b15e7a6b2b3b69cf05/external/local_config_apple_cc/BUILD:79:24: in cc_toolchain_config rule @@local_config_apple_cc//:darwin_arm64: 
Traceback (most recent call last):
        File "/private/var/tmp/_bazel_coryan/d7a3664b39ea78b15e7a6b2b3b69cf05/external/local_config_apple_cc/cc_toolchain_config.bzl", line 2465, column 58, in _impl
                enabled = xcode_support.is_xcode_at_least_version(xcode_config, "15.0.0"),
        File "/private/var/tmp/_bazel_coryan/d7a3664b39ea78b15e7a6b2b3b69cf05/external/build_bazel_apple_support/lib/xcode_support.bzl", line 35, column 13, in _is_xcode_at_least_version
                fail("Could not determine Xcode version at all. This likely means Xcode isn't available; " +
Error in fail: Could not determine Xcode version at all. This likely means Xcode isn't available; if you think this is a mistake, please file an issue.
ERROR: /private/var/tmp/_bazel_coryan/d7a3664b39ea78b15e7a6b2b3b69cf05/external/local_config_apple_cc/BUILD:79:24: Analysis of target '@@local_config_apple_cc//:darwin_arm64' failed
ERROR: Analysis of target '//:greeting' failed; build aborted: Analysis failed
INFO: Elapsed time: 9.443s, Critical Path: 0.01s
INFO: 1 process: 1 internal.
ERROR: Build did NOT complete successfully

---

Likely should just remove the rules_cc_toolchains() call from your WORKSPACE for now

I tried that too:

diff --git a/WORKSPACE.bazel b/WORKSPACE.bazel
index 681b370..f1e5616 100644
--- a/WORKSPACE.bazel
+++ b/WORKSPACE.bazel
@@ -42,8 +42,6 @@ load(
 
 rules_cc_dependencies()
 
-rules_cc_toolchains()
-
 load(
     "@build_bazel_rules_apple//apple:repositories.bzl",
     "apple_rules_dependencies",

and this:

diff --git a/WORKSPACE.bazel b/WORKSPACE.bazel
index 681b370..3855c1f 100644
--- a/WORKSPACE.bazel
+++ b/WORKSPACE.bazel
@@ -35,15 +35,6 @@ maybe(
     sha256 = "34c41bfb59cdaea29ac2df5a2fa79e5add609c71bb303b2ebb10985f93fa20e7",
 )
 
-load(
-    "@rules_cc//cc:repositories.bzl",
-    "rules_cc_dependencies",
-    "rules_cc_toolchains")
-
-rules_cc_dependencies()
-
-rules_cc_toolchains()
-
 load(
     "@build_bazel_rules_apple//apple:repositories.bzl",
     "apple_rules_dependencies",

Same results.

[keith]

Currently a fresh project without anything in the WORKSPACE should work with the BUILD file from your repro case, since the default toolchain in bazel should still apply, are you not finding that to be the case? The first failure you posted makes sense if you don't have Xcode installed and only have the command line tools

[coryan]

Currently a fresh project without anything in the WORKSPACE should work with the BUILD file from your repro case, since the default toolchain in bazel should still apply, are you not finding that to be the case?

I am trying to get Bazel v7 to work in a much larger project, that needs a WORKSPACE file for a number of dependencies (Protobuf, gRPC, Abseil, etc.). If you are curious the project is:

https://github.com/googleapis/google-cloud-cpp

Frankly, our project does not really need rules_apple, but our deps do, and they are loading versions of rules_apple that are too old for Bazel v7. The repro was the smallest thing that (a) still looks like how we use Bazel in the project (and its deps), and (b) still repros the problem.

The first failure you posted makes sense if you don't have Xcode installed and only have the command line tools

Is it necessary to install Xcode just to compile C++ programs? That seems sad, but I will gladly do it if that is what it takes. Is that documented anywhere in the Bazel v7 release notes? If it is, I probably missed it.

[coryan]

Is it necessary to install Xcode just to compile C++ programs?

It seems like it is not. Unfortunately one of our deps (gRPC) needs rules_apple even if you only are interested in compiling the C++ targets. Sigh.

[keith]

Is it necessary to install Xcode just to compile C++ programs?

no it's not, I just assumed you had rules_apple intentionally. The issue is the first toolchain that is registered and matches wins in this case. You can either manually register the built in bazel toolchain first, use my rules_cc patch linked above, or set common --repo_env=BAZEL_NO_APPLE_CPP_TOOLCHAIN=1 in your .bazelrc

[coryan]

Is it necessary to install Xcode just to compile C++ programs?

no it's not, I just assumed you had rules_apple intentionally.

nit: it is intentional. If I don't have it, then I cannot even start a build with Bazel v7 which depends on gRPC. . I am not going to use it directly, and most likely I won't even use targets in gRPC that need it. But what I would call the loading phase (I am not sure that is the right name) of Bazel fails without it.

The issue is the first toolchain that is registered and matches wins in this case. You can either manually register the built in bazel toolchain first, use my rules_cc patch linked above, or set common --repo_env=BAZEL_NO_APPLE_CPP_TOOLCHAIN=1 in your .bazelrc

BAZEL_NO_APPLE_CPP_TOOLCHAIN=1 was very useful, thanks. Most of our builds (googleapis/google-cloud-cpp#13274) are now working.

I think we can close this bug once bazelbuild/rules_cc#215 goes through.

[keith]

To summarize the issues here:

1. The first registered CC toolchain wins, bazel's default C++ toolchain is registered last
2. The apple_support toolchain is meant for building targets from rules_apple, and requires a full Xcode installation, not just the Xcode Command Line Tools (setting BAZEL_NO_APPLE_CPP_TOOLCHAIN=1 in the environment disables the registration of this toolchain all together)
3. gRPC depends on rules_apple / apple_support even if you don't use that part of gRPC
4. rules_cc registers an old broken copy of the default C++ bazel toolchain only when using WORKSPACE deps, not when using bzlmod (fix here Use bazel toolchain with workspace rules_cc#215)

[coryan]

@veblush FYI the previous comment seems relevant for the Bazel v7 work on gRPC.

[yjianpen]

Is it resolved? I met the exact same problem and it's really frustrating.

[iancha1992]

Is it resolved? I met the exact same problem and it's really frustrating.

cc: @comius @keith