Cached .d files contain absolute paths to system headers

[keith]

When building a cc_library on macOS using a remote cache, you can get build failures if the machines have different paths to their Xcode.app. The error ends up looking something like this:

INFO: Found 1 target...
ERROR: /private/var/tmp/_bazel_Scott/def6c6f164932cea6d66ed5d164b5ad3/external/StringsGenYams/BUILD.bazel:3:1: undeclared inclusion(s) in rule '@StringsGenYams//:CYaml':
this rule is missing dependency declarations for the following files included by 'external/StringsGenYams/Sources/CYaml/src/reader.c':
  '/Applications/Xcode-10.1.0.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include/stdlib.h'
  '/Applications/Xcode-10.1.0.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include/Availability.h'
  '/Applications/Xcode-10.1.0.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include/AvailabilityInternal.h'
  '/Applications/Xcode-10.1.0.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include/sys/cdefs.h'
  '/Applications/Xcode-10.1.0.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include/sys/_symbol_aliasing.h'
  '/Applications/Xcode-10.1.0.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include/sys/_posix_availability.h'
  '/Applications/Xcode-10.1.0.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include/_types.h'

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

You can see that the .d file contains host specific paths with:

bazel build examples/cpp:hello-lib

And then opening bazel-bin/examples/cpp/_objs/hello-lib/hello-lib.d. Here you can see paths like:

/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/string

But there's no real guarantee that the Xcode.app being used is at this same file path.

What operating system are you running Bazel on?

macOS

What's the output of bazel info release?

release 0.23.1

Have you found anything relevant by searching the web?

This appears to be related to this issue #1000 and the other issues linked to it.

I was hoping to work around this by disabling the .d file generation, there was an issue with that I filed here #7769

[keith]

Here's a possible fix for this issue #7783

[nlopezgi]

see also #7702
"the C++ rules are not well specified enough to safely share a remote cache between different linux distros, gcc and libc". I assume the same must be true for different mac machines.
"the recommended approach to avoid such problems is for a user to provide a unique key for each platform" so if 2 mac machines are sufficiently different (not sure differences in header dirs are 'sufficient'), you're probably better off if they dont share a cache.

[keith]

Thanks for the link! I don't think that the path to the same version of Xcode on 2 machines being different really warrants being "sufficiently different" so I feel like we should still find a way to share the cache between them

[keith]

For future reference about the resolution, these files still contain the paths, but when bazel maps them back if they're prefixed with /Applications/ they'll be handled correctly locally

[keith]

Note this also causes an issue during local development if you change Xcode paths without doing a clean --expunge

[thii]

@keith Do you mind pointing out where Bazel maps those header paths?

[keith]

bazel/tools/cpp/osx_cc_configure.bzl

Line 150 in e3e46e3

write_builtin_include_directory_paths(repository_ctx, cc, escaped_include_paths)

[thii]

Not sure if I'm understanding it correctly, but does it mean with BAZEL_IGNORE_SYSTEM_HEADERS_VERSIONS=1, Bazel won't remap them?

[keith]

I think things have changed a bit since I filed this issue so I may have been confused on your question. AFAIUI at this point there are 2 header related things that might affect this:

1. The crosstool generates a list of acceptable include paths, that it doesn't require to be explicitly defined, but won't error on. That is done here

   bazel/tools/cpp/osx_cc_configure.bzl

   Lines 31 to 50 in e3e46e3

   	def _get_escaped_xcode_cxx_inc_directories(repository_ctx, cc, xcode_toolchains):
   	"""Compute the list of default C++ include paths on Xcode-enabled darwin.
   	Args:
   	repository_ctx: The repository context.
   	cc: The default C++ compiler on the local system.
   	xcode_toolchains: A list containing the xcode toolchains available
   	Returns:
   	include_paths: A list of builtin include paths.
   	"""
   	# TODO(cparsons): Falling back to the default C++ compiler builtin include
   	# paths shouldn't be unnecessary once all actions are using xcrun.
   	include_dirs = get_escaped_cxx_inc_directories(repository_ctx, cc, "-xc++")
   	for toolchain in xcode_toolchains:
   	include_dirs.append(escape_string(toolchain.developer_dir))
   	# Assume that all paths that point to /Applications/ are built in include paths
   	include_dirs.append("/Applications/")
   	return include_dirs

   since the PR I linked above Add /Applications/ to cxx_builtin_include_directories #7783 this includes /Applications which solved my original issue
2. Bazel writes out the same list of headers, and uses those as an input to the build. IMO this fundamentally breaks usage with Xcode, since the Xcode abstraction with xcode-locator no longer applies, and the hard coded path to Xcode's headers are an input to the build. That's what I linked to above, and can be disabled with an env var

   bazel/tools/cpp/lib_cc_configure.bzl

   Lines 224 to 245 in e3e46e3

   	def write_builtin_include_directory_paths(repository_ctx, cc, directories, file_suffix = ""):
   	"""Generate output file named 'builtin_include_directory_paths' in the root of the repository."""
   	if get_env_var(repository_ctx, "BAZEL_IGNORE_SYSTEM_HEADERS_VERSIONS", "0", False) == "1":
   	repository_ctx.file(
   	"builtin_include_directory_paths" + file_suffix,
   	"""This file is generated by cc_configure and normally contains builtin include directories
   	that C++ compiler reported. But because BAZEL_IGNORE_SYSTEM_HEADERS_VERSIONS was set to 1,
   	header include directory paths are intentionally not put there.
   	""",
   	)
   	else:
   	repository_ctx.file(
   	"builtin_include_directory_paths" + file_suffix,
   	"""This file is generated by cc_configure and contains builtin include directories
   	that %s reported. This file is a dependency of every compilation action and
   	changes to it will be reflected in the action cache key. When some of these
   	paths change, Bazel will make sure to rerun the action, even though none of
   	declared action inputs or the action commandline changes.
   	%s
   	""" % (cc, "\n".join(directories)),
   	)