rust-analyzer fails to load proc-macros when loaded through ld.so

[jonhoo]

This is a somewhat niche problem, but I figured there should at least be a public record of it.

If rust-analyzer is executed through ld.so, it fails to load the shared library for proc-macros:

proc-macro: error while loading shared libraries: proc-macro: cannot open shared object file

You can reproduce this with:

cargo new --lib ra-ldso
cd ra-ldso
/lib/"ld-linux-$(uname -m).so.1" "$(which rust-analyzer)" analysis-stats .

Note that it's not even necessary to actually pass --library-path — invoking through ld.so is sufficient.

Loading through ld.so is primarily useful because it allows setting the shared library search path without setting LD_LIBRARY_PATH. This, in turn, is useful because LD_LIBRARY_PATH is "viral" — it also affects the search path for child processes, whereas ld.so --library-path does not. It's important to use a non-viral mechanism in nix-style applications where each program lives in its own runtime environment, and you want a way to set the runtime environment for a given binary, but not for any binaries that it may in turn execute, since those may themselves have their won runtime environments.

Based purely on a guess, I suspect this is related to the solution implemented through #12803, but hard to say for sure. Alternatively, it might be related to how rust-analyzer locates its own binary to execute the proc-macro-srv — if it uses argv0, then it'll now find ld.so rather than the true rust-analyzer!

rust-analyzer version: rust-analyzer 0.0.0 (ab068f120 2022-08-31)

rustc version: rustc 1.63.0 (4b91a6ea7 2022-08-08)

[Veykril]

Can you check your server logs and paste them here? They should have some more info potentially (I don't have a linux machine at hand)

[jonhoo]

$ RA_LOG=lsp_server=debug /lib/ld-linux-aarch64.so.1 "$(which rust-analyzer)" analysis-stats .
Failed to create perf counter: Operation not permitted (os error 1)
Failed to create perf counter: Operation not permitted (os error 1)
proc-macro: error while loading shared libraries: proc-macro: cannot open shared object file
Database loaded:     481.40ms (metadata 383.73ms; build 55.34ms)
Failed to create perf counter: Operation not permitted (os error 1)
  crates: 1, mods: 2, decls: 3, fns: 2
Item Collection:     3.90s
Failed to create perf counter: Operation not permitted (os error 1)
  exprs: 35, ??ty: 0 (0%), ?ty: 0 (0%), !ty: 0
Inference:           1.97s
Total:               5.86s

Or are you looking for something more than this?

[Veykril]

Can you set the RA_LOG to warn,rust_analyzer::reload=debug,project_model::build_scripts=info? I think that should have the important things

[jonhoo]

$ env RA_LOG=warn,rust_analyzer::reload=debug,project_model::build_scripts=info /lib/ld-linux-aarch64.so.1 $(which rust-analyzer) analysis-stats .
Failed to create perf counter: Operation not permitted (os error 1)
Failed to create perf counter: Operation not permitted (os error 1)
[INFO project_model::build_scripts] Running build scripts: "cargo" "check" "--quiet" "--workspace" "--message-format=json" "--all-targets"
proc-macro: error while loading shared libraries: proc-macro: cannot open shared object file
Database loaded:     486.08ms (metadata 386.89ms; build 55.84ms)
Failed to create perf counter: Operation not permitted (os error 1)
  crates: 1, mods: 2, decls: 3, fns: 2
Item Collection:     3.96s
Failed to create perf counter: Operation not permitted (os error 1)
  exprs: 35, ??ty: 0 (0%), ?ty: 0 (0%), !ty: 0
Inference:           2.17s
Total:               6.13s

[jonhoo]

And in case it's useful:

$ cargo check --quiet --workspace --message-format=json
{"reason":"compiler-artifact","package_id":"ra-test 0.1.0 (path+file:///local/home/jongje/ra-test)","manifest_path":"/local/home/jongje/ra-test/Cargo.toml","target":{"kind":["lib"],"crate_types":["lib"],"name":"ra-test","src_path":"/local/home/jongje/ra-test/src/lib.rs","edition":"2021","doc":true,"doctest":true,"test":true},"profile":{"opt_level":"0","debuginfo":2,"debug_assertions":true,"overflow_checks":true,"test":false},"features":[],"filenames":["/local/home/jongje/ra-test/target/debug/deps/libra_test-6c441a19cebcf54a.rmeta"],"executable":null,"fresh":true}
{"reason":"build-finished","success":true}

[Veykril]

Oh right, analysis-stats is not running the server. Ye so I think your assumption might be right, analysis-stats is always running the "current exe" with proc-macro as the arg unlike the server that now uses the sysroot if possible

rust-analyzer/crates/rust-analyzer/src/cli/load_cargo.rs

Lines 60 to 66 in 643c3a5

	let proc_macro_client = if load_config.with_proc_macro {
	let path = AbsPathBuf::assert(std::env::current_exe()?);
	Ok(ProcMacroServer::spawn(path, &["proc-macro"]).unwrap())
	} else {
	Err("proc macro server not started".to_owned())
	};

Though #12803 is unrelated to that as this was already the behavior prior to that

[Veykril]

So when using r-a as the server this should work fine (as long as the rust sysroot is recent enough), otherwise this fails with the same problem as we fall back to the current exe.

There is a config (that analysis-stats doesn't use since it isn't configurable) to set the proc-macro server path, though that is a bit odd as well as it currently works relative to the workspace you have opened (we should fix that, but haven't yet).

[jonhoo]

Hmm, I'm seeing exactly the same failure when using rust-analyzer through ld.so just as an LSP server (with neovim LSP as the client)..

[bjorn3]

When rust-analyzer tries to load proc macros, it does so by dlopen'ing them in a newly spawned process. This process is not created by invoking the same ld.so as you did for the main rust-analyzer process. Also the libc version needs to match (or be newer than) what rustc compiled the proc macro against.

if it uses argv0, then it'll now find ld.so rather than the true rust-analyzer!

It does either that or /proc/self/exe (through std::env::current_exe()).

By the way why aren't you using patchelf?

[jonhoo]

When rust-analyzer tries to load proc macros, it does so by dlopen'ing them in a newly spawned process. This process is not created by invoking the same ld.so as you did for the main rust-analyzer process.

That doesn't explain the problem I don't think, since rust-analyzer fails when invoked this way even if no arguments are provided to ld.so. That is, I'm not actually overriding anything.

Also the libc version needs to match (or be newer than) what rustc compiled the proc macro against.

Yep, that should be the case here.

if it uses argv0, then it'll now find ld.so rather than the true rust-analyzer!

It does either that or /proc/self/exe (through std::env::current_exe()).

Actually, looking at it again, /proc/self/exe will point to ld.so in this situation. It looks like "fixing" argv0 to point to rust-analyzer doesn't fix the problem:

$ sh -c "exec -a $(which rust-analyzer) /lib/ld-linux-aarch64.so.1 $(which rust-analyzer) analysis-stats ."
...
proc-macro: error while loading shared libraries: proc-macro: cannot open shared object file

Which suggests that rust-analyzer is indeed relying just on /proc/self/exe (which leads it astray with ld.so).

By the way why aren't you using patchelf?

patchelf would set RPATH, which either requires an absolute path (which would make the binary impossible to run on other hosts) or requires using $ORIGIN, which resolves symlinks before relative paths, which in turn breaks things in my setup where many of the binaries (and shared libraries) will be symlinked into a common directory to construct the runtime environment.

[jonhoo]

For some related context, see also rust-lang/cargo#10115 and rust-lang/cargo#10113.

[bjorn3]

I believe the NixOS version of rustup automatically runs patchelf on all binaries it downloads. That should be fine, right? It doesn't matter that the path is absolute as you won't copy the toolchain to another machine anyway. There is a rustup component for rust-analyzer and even a rustup component for just the proc macro server of rust-analyzer (not sure if it is available on stable already. it is available on beta) in case you want to keep using non-rustup versions of rust-analyzer. It should automatically load the proc macro server from the toolchain if available.

[jonhoo]

Ah, so, to be clear, while the system I'm working within (Amazon's build system) shares some similarities with Nix, it is not Nix. Nix has the advantage that paths are the same across hosts, so setting absolute paths in RPATH is fine. We do not have that luxury sadly. And we do want to be able to use the same runtime environments across different hosts.

There's also no rustup involved here — the value from $(which rust-analyzer) is a path to a standalone rust-analyzer binary (from cargo xtask install).

[Veykril]

Out of curiosity, what rust version are you working with right now? (Should've been clearer on this) If you are on stable, then yes the rust-analyzer server itself will also fail as the sysroot change hasnt landed there yet, that is currently only on recent nightlies

[jonhoo]

Yes, this is with stable Rust, but with rust-analyzer build directly from the latest git release tag.

[Veykril]

Ye then this will occur with the rust-analyzer server as well. If you have a recent nightly lying around, you can try it with that to check if that fixes it (I believe it should), in which case this should fix itself with the next release of stable as well.

Though I assume as this is something with Amazon's build system here, you are stuck on a certain version for a bit. I assume you want to be able to run the proc-macro server via ld.so as well ideally? Otherwise I'd say we just need to fix the proc-macro server config so that it isn't relative to the workspace.

[jonhoo]

Same thing happens with most recent nightly (set through a rustup override for the current directory):

$ rustc -V
rustc 1.65.0-nightly (9243168fa 2022-08-31)
$ sh -c "exec -a $(which rust-analyzer) /lib/ld-linux-aarch64.so.1 $(which rust-analyzer) analysis-stats ."
...
proc-macro: error while loading shared libraries: proc-macro: cannot open shared object file

[Veykril]

That is odd, if r-a tries to use the server in the rust sysroot it shouldn't be passing any arguments at all... So I guess it's not finding the sysroot here for some reason ...
Well I guess I'll look into the fixing the path config, that way you can test this by specifying a path for the server directly instead of it falling back to the current exe

[jonhoo]

Happy to help test more if there's anything that'd be helpful!

[Veykril]

Oh wait, can you try this (nightly toolchain) with the actual rust-analyzer server (invoked by your text editor), not analysis-stats? analysis-stats never uses the sysroot

[jonhoo]

Interesting, so that actually fails in a different way:

$ rustc -V
rustc 1.65.0-nightly (9243168fa 2022-08-31)
$ which rust-analyzer
/home/jongje/bin/rust-analyzer
$ cat ~/bin/rust-analyzer
#!/bin/bash
exec -a ~/bin/rust-analyzer.upstream /lib/ld-linux-aarch64.so.1 ~/bin/rust-analyzer.upstream "$@"
$ ~/bin/rust-analyzer.upstream --version
rust-analyzer 0.0.0 (ab068f120 2022-08-31)
$ nvim src/lib.rs
$ cat ~/.cache/nvim/lsp.log
[START][2022-09-01 18:28:22] LSP logging initiated
[ERROR][2022-09-01 18:28:24] .../vim/lsp/rpc.lua:420    "rpc"   "rust-analyzer" "stderr"        "[ERROR rust_analyzer::lsp_utils] failed to run build scripts\n\nerror: failed to run `rustc` to learn about target-specific information\n\nCaused by:\n  process didn't exit successfully: `/usr/lib64/ld-2.26.so rustc - --crate-name ___ --print=file-names --crate-type bin --crate-type rlib --crate-type dylib --crate-type cdylib --crate-type staticlib --crate-type proc-macro --print=sysroot --print=cfg` (exit status: 127)\n  --- stderr\n  rustc: error while loading shared libraries: rustc: cannot open shared object file\n\n\n"
$ rust-analyzer analysis-stats .
Failed to create perf counter: Operation not permitted (os error 1)
Failed to create perf counter: Operation not permitted (os error 1)
proc-macro: error while loading shared libraries: proc-macro: cannot open shared object file
Database loaded:     596.27ms (metadata 496.92ms; build 55.37ms)
Failed to create perf counter: Operation not permitted (os error 1)
  crates: 1, mods: 2, decls: 3, fns: 2
Item Collection:     4.03s
Failed to create perf counter: Operation not permitted (os error 1)
  exprs: 35, ??ty: 0 (0%), ?ty: 0 (0%), !ty: 0
Inference:           1.96s
Total:               5.99s

That is:

failed to run build scripts

error: failed to run `rustc` to learn about target-specific information

Caused by:
  process didn't exit successfully: `/usr/lib64/ld-2.26.so rustc - --crate-name ___ --print=file-names --crate-type bin --crate-type rlib --crate-type dylib --crate-type cdylib --crate-type staticlib --crate-type proc-macro --print=sysroot --print=cfg` (exit status: 127)
  --- stderr
  rustc: error while loading shared libraries: rustc: cannot open shared object file

[jonhoo]

I'm guessing ld-2.26.so there was intended to be cargo? Which is probably then related to rust-lang/cargo#10113 / rust-lang/cargo#10115 / rust-lang/cargo#10119.

[bjorn3]

rustc: error while loading shared libraries: rustc: cannot open shared object file

Two things: First rustc uses std::env::current_exe() to get the location of the sysroot, which would fail due to /proc/self/exe pointing to the wrong location. Second it seems that in this case the full path to the rustc executable doesn't get passed, which I think makes ld.so fail with this error. Try setting RUSTC to the full path to rustc.

I'm guessing ld-2.26.so there was intended to be cargo? Which is probably then related to rust-lang/cargo#10113 / rust-lang/cargo#10115 / rust-lang/cargo#10119.

No, this is cargo invoking rustc, not rust-analyzer invoking cargo.

[Veykril]

I believe the RUSTC_WRAPPER will be causing problems as well ...

rust-analyzer/crates/project-model/src/build_scripts.rs

Lines 107 to 113 in c4445e4

	if config.wrap_rustc_in_build_scripts {
	// Setup RUSTC_WRAPPER to point to `rust-analyzer` binary itself. We use
	// that to compile only proc macros and build scripts during the initial
	// `cargo check`.
	let myself = std::env::current_exe()?;
	cmd.env("RUSTC_WRAPPER", myself);
	cmd.env("RA_RUSTC_WRAPPER", "1");

This is the only other place where we use current_exe here though that is relevant. Though you can disable the wrapper (at the cost of starting times and the need that your project has to be buildable when r-a starts)

[jonhoo]

Second it seems that in this case the full path to the rustc executable doesn't get passed, which I think makes ld.so fail with this error. Try setting RUSTC to the full path to rustc.

Yep, I can confirm that with

$ env RUSTC=$(which rustc) nvim src/lib.rs

(and while still on nightly), rust-analyzer works just fine with the ld.so wrapper (at least as far as I can tell so far)!