Issues with 'version `GLIBC_2.32' not found' when running clippy on CI

[archseer]

# flake.nix

{
  description = "chainlink-terra";

  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
    flake-utils.url = "github:numtide/flake-utils";
    rust-overlay.url = "github:oxalica/rust-overlay";
  };

  outputs = inputs@{ self, nixpkgs, rust-overlay, flake-utils, ... }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = import nixpkgs { inherit system; overlays = [ rust-overlay.overlay ]; };
      in rec {
        devShell = pkgs.callPackage ./shell.nix {};
      });
}

# shell.nix

{ stdenv, pkgs, lib }:

pkgs.mkShell {
  nativeBuildInputs = with pkgs; [
    (rust-bin.stable.latest.default.override {
      extensions = ["rust-src"];
      targets = [
        "x86_64-unknown-linux-gnu"
        "wasm32-unknown-unknown"
      ];
    })
    cargo-generate
    cargo-tarpaulin
  ];
  RUST_BACKTRACE = "1";
}

When running nix develop -c cargo clippy --all-targets -- -D warnings it fails during proc macro checking:

    Checking thiserror v1.0.29
error: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /home/runner/work/foo/target/debug/deps/libthiserror_impl-22080d5093a0fbd6.so)
   --> /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/thiserror-1.0.29/src/lib.rs:213:9
    |
213 | pub use thiserror_impl::*;
    |         ^^^^^^^^^^^^^^

However compiling, running tests, nix develop -c cargo fmt or nix develop -c cargo build --release --target wasm32-unknown-unknown all works fine.

Possibly related: https://internals.rust-lang.org/t/bootstrap-fails-with-linker-error-while-running-on-non-nixos-nix-environment-wsl2-ubuntu/15066/2

[oxalica]

Cannot reproduce on a empty project with a derive(thiserror::Error) struct.

Please check these:

1. nix flake update.
2. Are you using any cargo config like .cargo/config*, ~/.cargo/config* or $CARGO_HOME? Please disable them.
3. Delete target directory and run nix develop --ignore-environment -c cargo clippy --all-targets -- -D warnings

If possible, please also provide the (minimized) project's Cargo.toml and source code.

[archseer]

Tried these:

1. I ran update.
2. We are using a .cargo/config file but it only contains a few command aliases.
3. I rearranged cargo clippy to run first and ignore the env as well as made sure there's no cache set up.

I also can't reproduce locally on NixOS, only on github actions (which are Nix on Ubuntu, not NixOS, that's why maybe the thread above is relevant).

I'll see if I can provide a minimized reproduction case, in general it seems that build or test or check cases work and run fine. It only fails with clippy on the check step, not the compilation (of thiserror_impl). So it could be either an issue in clippy's resolution or a problem with clippy not being fully patched.

[oxalica]

We are using a .cargo/config file but it only contains a few command aliases.

I mainly concern about the linker and rustflags fields since this issue is related to linking.

The error is like that when compiling proc-macro crate thiserror, it is linked to the glibc outside nix store. It may be caused by system gcc being used to link, instead of nix stdenv's gcc.

You can also try to add gcc to nativeBuildInputs (though it should be automatically included), and add nix develop -i -c bash -c 'type gcc' in CI before clippy. It should print a path starting with /nix/store.

[archseer]

I think it must be an issue specific to clippy:

I've added gcc to nativeBuildInputs:

gcc is /nix/store/s5hkav7whndbfz0szshpb46h4idqdq9a-gcc-wrapper-10.3.0/bin/gcc

clippy still fails the same way.

The comment here mentions something about bootstrap being patched on NixOS, but not Nix on other Linux platforms: https://internals.rust-lang.org/t/bootstrap-fails-with-linker-error-while-running-on-non-nixos-nix-environment-wsl2-ubuntu/15066/7

[abbec]

I am also getting this. It seems that the rustc binary is not properly patched and uses system libc:

❯ which rustc
/nix/store/lkmigamdvgjb0aipx782qrzizdp02w5j-rust-default-1.56.1/bin/rustc

❯ ldd $(which rustc)
        linux-vdso.so.1 (0x00007fff878d0000)
        librustc_driver-1a7f7f6d0c5586ea.so => /nix/store/dh285vqm29i487zfi57csbdjww7agyvz-rustc-1.56.1/lib/librustc_driver-1a7f7f6d0c5586ea.so (0x00007f35718d3000)
        libstd-65a28bf1738424c0.so => /nix/store/dh285vqm29i487zfi57csbdjww7agyvz-rustc-1.56.1/lib/libstd-65a28bf1738424c0.so (0x00007f3571527000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f35714ef000)
        librt.so.1 => /lib64/librt.so.1 (0x00007f35714e4000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f35714dd000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f357130e000)
        libLLVM-13-rust-1.56.1-stable.so => /nix/store/dh285vqm29i487zfi57csbdjww7agyvz-rustc-1.56.1/lib/libLLVM-13-rust-1.56.1-stable.so (0x00007f356bf14000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f356bef9000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f356bdb5000)
        /nix/store/jsp3h3wpzc842j0rz61m5ly71ak6qgdn-glibc-2.32-54/lib/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x00007f35759df000)
        libz.so.1 => /nix/store/ds491f6b5pdk3xxnc2w103asyz1y4cfc-zlib-1.2.11/lib/libz.so.1 (0x00007f356bd98000)

[abbec]

Compared to the rustc in nixpkgs:

❯ ldd $(which rustc)
        linux-vdso.so.1 (0x00007ffe0dbe5000)
        librustc_driver-91277fe408b8ac49.so => /nix/store/sw7kr9clh3y3rpghgh50j023nbnwqz9i-rustc-1.55.0/bin/../lib/librustc_driver-91277fe408b8ac49.so (0x00007ff2674e7000)
        libstd-6556354a51bce6a2.so => /nix/store/sw7kr9clh3y3rpghgh50j023nbnwqz9i-rustc-1.55.0/bin/../lib/libstd-6556354a51bce6a2.so (0x00007ff26732b000)
        libc.so.6 => /nix/store/mij848h2x5wiqkwhg027byvmf9x3gx7y-glibc-2.33-50/lib/libc.so.6 (0x00007ff267166000)
        libLLVM-12.so => /nix/store/a96n4dxdfk76sxmx8jmdiy3l2yhqzym4-llvm-12.0.1-lib/lib/libLLVM-12.so (0x00007ff260cb4000)
        libstdc++.so.6 => /nix/store/7fv9v6mnlkb4ddf9kz1snknbvbfbcbx0-gcc-10.3.0-lib/lib/libstdc++.so.6 (0x00007ff260add000)
        libgcc_s.so.1 => /nix/store/mij848h2x5wiqkwhg027byvmf9x3gx7y-glibc-2.33-50/lib/libgcc_s.so.1 (0x00007ff260ac3000)
        libpthread.so.0 => /nix/store/mij848h2x5wiqkwhg027byvmf9x3gx7y-glibc-2.33-50/lib/libpthread.so.0 (0x00007ff260aa3000)
        libm.so.6 => /nix/store/mij848h2x5wiqkwhg027byvmf9x3gx7y-glibc-2.33-50/lib/libm.so.6 (0x00007ff260962000)
        libdl.so.2 => /nix/store/mij848h2x5wiqkwhg027byvmf9x3gx7y-glibc-2.33-50/lib/libdl.so.2 (0x00007ff26095d000)
        /nix/store/mij848h2x5wiqkwhg027byvmf9x3gx7y-glibc-2.33-50/lib/ld-linux-x86-64.so.2 => /nix/store/mij848h2x5wiqkwhg027byvmf9x3gx7y-glibc-2.33-50/lib64/ld-linux-x86-64.so.2 (0x00007ff26af14000)
        libffi.so.8 => /nix/store/2z8hacx9dphisbf7syd9hvkiw4578r23-libffi-3.4.2/lib/libffi.so.8 (0x00007ff26094e000)
        librt.so.1 => /nix/store/mij848h2x5wiqkwhg027byvmf9x3gx7y-glibc-2.33-50/lib/librt.so.1 (0x00007ff260943000)
        libz.so.1 => /nix/store/c5giadpm71mvla6202mhz32x9criabna-zlib-1.2.11/lib/libz.so.1 (0x00007ff260926000)
        libncursesw.so.6 => /nix/store/jzjqfff4cldlm2wpld313a2s1v6r3ycb-ncurses-6.2/lib/libncursesw.so.6 (0x00007ff2608b4000)
        libxml2.so.2 => /nix/store/jjl2npsfbcwbd80c1ap6ls40s7hak53s-libxml2-2.9.12/lib/libxml2.so.2 (0x00007ff260746000)

[abbec]

More specifically, the (working) Nixpkgs one has rpath like:

$ readelf -d $(which rustc) | head -20

Dynamic section at offset 0x2da8 contains 28 entries:
  Tag        Type                         Name/Value
...
 0x000000000000001d (RUNPATH)            Library runpath: [$ORIGIN/../lib:/nix/store/sw7kr9clh3y3rpghgh50j023nbnwqz9i-rustc-1.55.0/lib:/nix/store/mij848h2x5wiqkwhg027byvmf9x3gx7y-glibc-2.33-50/lib]
...

whereas the one from this overlay has

readelf -d $(which rustc) | head -20

Dynamic section at offset 0x61bc0 contains 31 entries:
  Tag        Type                         Name/Value
...
 0x000000000000001d (RUNPATH)            Library runpath: [/nix/store/dh285vqm29i487zfi57csbdjww7agyvz-rustc-1.56.1/lib]
...

where glibc is missing from the rpath.

[ajeetdsouza]

I'm getting this error too on cargo clippy when using proc macros. I'm using Nix on Ubuntu 20.04.3 on WSL2.

$ ldd --version
ldd (Ubuntu GLIBC 2.31-0ubuntu9.2) 2.31

I tried Mozilla's overlay as well as using the cargo package from nixpkgs-unstable directly, and I'm getting the same error there as well.

[oxalica]

More specifically, the (working) Nixpkgs one has rpath like:
...
where glibc is missing from the rpath.

That's unrelated. glibc is set as interpreter, so it automatically adds to RPATH. Even if you manually add it, it would still be stripped as unnecessary paths in the "shrinking RPATH" step.

I cannot reproduce it on my Arch VM, but do reproduce it on GitHub CI. The log said cargo is actually invoking cargo-driver from rustup, outside environment of nix. It's more weird that nix-shell --pure, which unset all environment variables outside, still doesn't help. I can confirm that cargo, cargo-clippy and clippy-driver are all reachable from PATH in nix-shell environment with type. Not sure what magic happens in here.

CI log, (using nightly default, but still it try to invoke rustup's stable clippy-driver)
https://github.com/oxalica/rust-overlay/runs/4398112100?check_suite_focus=true#step:5:22

[yanganto]

I also run into this problem. I can confirm that this only happened on Ubuntu with nix on CI, and works good on NixOS.

[oxalica]

Okay, I got it. When cargo searching for subcommands, it looks up $CARGO_HOME/bin/<cmd> first, then ~/.cargo/bin/<cmd>, finally through $PATH. This order is hard-coded in

1. https://github.com/rust-lang/cargo/blob/rust-1.57.0/src/bin/cargo/main.rs#L196-L202
2. https://github.com/brson/home/blob/v0.5.1/src/lib.rs#L108-L130

I don't think rust-overlay could do anything here, since we are building from upstream binaries and cannot patch the source.

For anyone running into this issue, please run rm -rf ~/.cargo/bin to get rid of non-nix binaries in CI environment.

[ajeetdsouza]

@oxalica I don't have a very thorough understanding of Nix, but would it be possible to somehow use $CARGO_HOME in my Nix environment to prioritize the correct binary?

[oxalica]

would it be possible to somehow use $CARGO_HOME in my Nix environment to prioritize the correct binary?

Yes and no. You can set it to rust-bin.<whatever-version>.default so it can pick up binaries from ours first. But if so, you are not able to use any 3rd party crates. cargo will try to write crate index and sources into $CARGO_HOME/registry and fail due to read-only /nix/store.

Personally, I hate the design of CARGO_HOME since everything related or unrelated is mixed and squeezed into a single directory, and you are not able to fine-tune each one, only can move the hodgepodge together around.

[ajeetdsouza]

Makes sense. Here's how I fixed the problem on my system:

mv ~/.cargo/bin/* ~/.local/bin

I'll have to do this every time I upgrade, I imagine, but it solves the problem for now.

@oxalica if there's any way the Cargo team can fix this (say, an environment variable like $CARGO_BIN), perhaps it would make sense to create a feature request on their tracker?

[brendanzab]

For future reference, when running into this issue it appeared to me as the following error :

error: process didn't exit successfully: `rustc -vV` (exit status: 127)
--- stderr
rustc: error while loading shared libraries: libz.so.1: cannot open shared object file: No such file or directory

I searched for this error message in the issues but couldn’t see something that fit. Based on what I read in NixOS/nixpkgs#92946, I added the following to my devShell:

{
  LD_LIBRARY_PATH =
    # Clippy appears to requires zlib on linux
    lib.optionals pkgs.stdenv.isLinux
      (lib.makeLibraryPath [ pkgs.zlib ]);
}

This then uncovered the GLIBC_2.32 error:

 error: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /home/runner/work/fathom/fathom/target/debug

This led me to this issue! Based on the above discussion, I removed my devShell hack above, and added the following fix to my Github workflow which seemed to do the trick:

# This is a workaround for https://github.com/oxalica/rust-overlay/issues/54,
# avoiding link errors when running cargo commands with `nix develop`.
- name: Remove existing binaries from ~/.cargo/bin
  run: rm --recursive --force --verbose ~/.cargo/bin

Hope this helps anyone in the future!

if there's any way the Cargo team can fix this (say, an environment variable like $CARGO_BIN), perhaps it would make sense to create a feature request on their tracker?

Yet again wishing for rust-lang/rfcs#1615 I guess? 😅