No way to specify linker arguments via rustflags for dependency build scripts with --target

[mqudsi]

Problem

Background, but note that I already know how to work around this for me but am posting for want of a more general solution: I'm using rust on an unsupported macOS target (toolchain installed with macports) for $reasons. Due to missing system symbols like _getentropy() and _clock_gettime, the rust toolchain is built additionally linked against /opt/local/lib/libMacportsLegacySystem.B.dylib which provides these symbols that the standard library now uses since support for older macOS releases was removed.

Compiling a project with transitive dependencies that use std functionality that pulls in a reference to these symbols requires that the cargo be executed as env RUSTFLAGS='-Clink-args=-Wl,/opt/local/lib/libMacportsLegacySystem.B.dylib' cargo build and works great. No problems.

However, due to the logic that cargo uses internally to determine when and where rustflags are forwarded to rustc, this does not work if cargo is executed with an explicit --target, even when the target matches the host (e.g. x86_64-apple-darwin in this case).

I have tried the following methods:

• env RUSTFLAGS='...' cargo build --target x86_64-apple-darwin
• env CARGO_BUILD_RUSTFLAGS='...' cargo build --target x86_64-apple-darwin
• env CARGO_TARGET_X86_64_APPLE_DARWIN_RUSTFLAGS='...' cargo build --target x86_64-apple-darwin
• Specifying target.x86_64-apple-darwin.rustflags in a newly created .config/cargo.toml (to emphasize that there was no previous such file masking the above)

In all these cases, the provided rust flags are not passed when the linker is invoked in dependency builds (e.g. pcre2-sys fails to compile with errors about a missing _getentropy() symbol).

Debugging with env CARGO_LOG=trace reveals that the flags are picked up, and they're even used for some of the rustc invocations, but critically, they're ignored when it comes to build scripts (which is predominantly where external linker invocations will come in).

I can understand why "normal" RUSTFLAGS are not forwarded to all rustc invocations when --target is used, but I would proffer that the specific combination of CARGO_TARGET_<TRIPLE>_RUSTFLAGS with a matching --target <triple> should be forwarded the same way that RUSTFLAGS is when no --target is passed. (Note here that the project builds successfully when using RUSTFLAGS and no --target, with the correct -Clink-args being forwarded throughout.)

Note also that I am requesting that a solution be possible via the usage of environment variables only, without needing to specify any values in .cargo/config.toml so that this can be used in external build systems. (Indeed, the problem here is that I am calling cargo from a build system that determines the target and always passes --target <triple> to cargo.)

(For the record, I am not reporting this as a "I can't build on this unsupported platform" bug but rather as a "there doesn't seem to be any way to influence the linker behavior of all transitive dependencies in one go with an environment variable" issue only when building for a specific --target triple.)

cc #9743, which identified the issue but ended up being treated as a symptom of the OP's problem rather than the problem itself, and was closed accordingly.

Steps

No response

Possible Solution(s)

• Always pass CARGO_TARGET_<TRIPLE>_RUSTFLAGS to build scripts when an explicitly provided, matching cargo build --target triple is used
• Extract at least linker-specific flags like -Clink-args and other -Clink... flags and pass those to build scripts where currently no rustflags are forwarded

Notes

No response

Version

cargo 1.78.0
release: 1.78.0
host: x86_64-apple-darwin
libgit2: 1.7.2 (sys:0.18.2 system)
libcurl: 8.7.1 (sys:0.4.72+curl-8.6.0 system ssl:OpenSSL/3.1.5)
os: Mac OS 10.10.5 [64-bit]

[mqudsi]

Should this be considered a duplicate of #4423?

[linyihai]

When not using --target, this has a consequence that Cargo will share your dependencies with build scripts and proc macros. RUSTFLAGS will be shared with every rustc invocation. With the --target flag, build scripts and proc macros are built separately (for the host architecture), and do not share RUSTFLAGS.

I found the section in Cargo book

Should this be considered a duplicate of #4423?

Very likely

[mqudsi]

@epage #4423 is a broad issue that is much more difficult to solve without introducing some major changes; would you consider entertaining a PR that forwarded rustflags to build scripts specifically and only for the case where host triple == explicit target triple? That would make cargo build equal to cargo build --target <host triple>, which, I think, is fairly uncontroversial?

[epage]

Sounds like what you are describing is target-applies-to-host

[mqudsi]

If I've understood that link correctly, what I want is actually the opposite of that, as target-applies-to-host already defaults to true and is the reason why RUSTFLAGS is being applied to everything (which I want) except the build scripts (which I wish it actually extended to). What I am asking for is a true "target-applies-to-host" that applies to everything, everywhere (whereas the current target-applies-to-host option is intended to expose the target-applies-to-host=false toggle, which is expressly the opposite of what I am seeking).

It does seem that -Zhost-config would work here, but that requires nightly :(

For context, I am on a no-longer-supported OS X version for $reasons, and since std now assumes that methods like getentropy() are always present on the host system (not true for older macOS/OS X releases), apps using that functionality end up with missing symbol linker errors. The workaround here is to link against the MacPorts "LegacySupport" library that provides shims for these, and to work around an assertion failure (internal bug) in the latest available ld for OS X, I need to compile anything that runs on the host machine with -Clink-arg=-fuse-ld=lld -Clink-arg=-Wl,/opt/local/lib/libMacportsLegacySystem.B.dylib.

For normal builds, merely setting RUSTFLAGS to the needed does the job, but the problem is when compiling with --target $host_target_triple that there is no combination of .cargo/config.toml or environment variable values that will pass these required linker flags to cc when invoking build scripts in transitive dependencies (without patching their build.rs to force these to be emitted). It seems from the link you shared that -Zhost-config or [host] in .cargo/config.toml would theoretically do what I need, but as I can't arbitrarily install a nightly toolchain to use these flags.

For further context, the issue is with "packaged" rust projects that cannot be installed normally via cargo install and have an external build system that further provides specific functionality required to create installable artifacts. In this specific case, I'm a fish-shell core dev and we've ported our codebase from C++ to rust but we still need to use CMake to perform the install step because we need to install support libraries to $PREFIX/share/fish/, and we would certainly prefer to not have to embed megabytes of support files in the binary just so that we can deploy them on first run as being cargo install deployable (without an external build system) would require. The further issue is that CMake invokes cargo with an explicit --target in all cases, even when target matches the host (mainly so that the deployment code is the same regardless of cross-compilation or not), so the RUSTFLAGS that we set in the CMake environment and pass to Cargo are not being used for the build scripts.

But the problem is obviously more general than that; the root problem is a general conflation of host and target systems for RUSTFLAGS and the "workaround" here was to completely ignore RUSTFLAGS for build scripts when --target was specified, but this (to my eyes) was not a fully studied option and has ramifications like the ones above that can actually leave one completely unable to target the host system.

So again, my suggestion would be that today a PR to patch cargo to either forward RUSTFLAGS to build scripts as well specifically when host matches target, or (less discoverable but I can't see any problem with this suggestion) at the very least, CARGO_TARGET_<TRIPLE>_RUSTFLAGS when specifically building for the same target triple (and possibly when target matches host if you want to be truly paranoid), to build scripts (along with everything else). The current status quo of "RUSTFLAGS applies everywhere, even to build scripts, but if --target is set and even if target matches host, it doesn't get passed to build scripts and there's absolutely nothing you can say or do about it" is, in my opinion and from where I'm standing, hard to defend as it is clearly a hack and leaves some cases completely unaddressed (and worse, unaddressable).

[weihanglo]

So again, my suggestion would be that today a PR to patch cargo to either forward RUSTFLAGS to build scripts as well specifically when host matches target, or (less discoverable but I can't see any problem with this suggestion) at the very least, CARGO_TARGET_<TRIPLE>_RUSTFLAGS when specifically building for the same target triple (and possibly when target matches host if you want to be truly paranoid), to build scripts (along with everything else).

This might break existing builds, for example, when using address sanitizer.

[mqudsi]

for example, when using address sanitizer.

Interesting case, I confess I didn't consider it... but I'm not sure it's so cut and dry. I set up the ASAN CI profile for fish, and so speaking from experience, I can imagine scenarios where passing RUSTFLAGS to build scripts could both break or actually be required for the correct application of ASAN. It's easy to see why you wouldn't want your build script to be run under ASAN (it'll mess with its output and you might have memory leaks that trigger an ASAN abort causing your builds to fail, among others) but there are also inverse cases (though I think they would tend to be non-idiomatic). CFLAGS is passed to build.rs and that's how you can usually get -fsanitizer=asan to be applied to C/C++ dependencies built by build.rs, but if you are building rust dependencies "out-of-band" (e.g. perhaps they don't use cargo or perhaps they're transitive dependencies of a C/C++ library you are building in build.rs), then you'll actually end up with a mix of ASAN-enabled and non-ASAN-enabled object files! (It's also "weird" that CFLAGS is forwarded to build.rs but RUSTFLAGS isn't, strictly from a philosophical purity POV.)

[mqudsi]

Is RUSTFLAGS currently not forwarded both when building build.rs and when executing it?

[weihanglo]

Is RUSTFLAGS currently not forwarded both when building build.rs and when executing it?

It depends on how you configure it.

Cargo does respect RUSTFLAGS from the environment when building a package. Mind the precedence in the doc and aforementioned --target behavior.

When running a build script, Cargo sets CARGO_ENCODED_RUSTFLAGS and removes RUSTFLAGS.

CARGO_ENCODED_RUSTFLAGS — extra flags that Cargo invokes rustc with, separated by a 0x1f character (ASCII Unit Separator). See build.rustflags. Note that since Rust 1.55, RUSTFLAGS is removed from the environment; scripts should use CARGO_ENCODED_RUSTFLAGS instead.

[mqudsi]

@weihanglo sorry, I meant specifically in the explicit --target case, not in general.

[weihanglo]

CARGO_ENCODED_RUSTFLAGS is rustflags for the package a build script belongs to, not for building build script itself. So, it is always available for a running build script. For example, the build script execution would get a CARGO_ENCODED_RUSTFLAGS="--verbose" when running the command below:

cargo build --target wasm32-unknown-unknown --config 'target.wasm32-unknown-unknown.rustflags="--verbose"

While compiling the build script itself wouldn't.