Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cargo test miri fails on macos 10.15.4 intermittent IO error while iterating directory #1285

Closed
Firstyear opened this issue Mar 31, 2020 · 10 comments
Closed

Cargo test miri fails on macos 10.15.4 intermittent IO error while iterating directory #1285

Firstyear opened this issue Mar 31, 2020 · 10 comments
Labels
A-cargo Area: affects the cargo wrapper (cargo miri) C-support Category: Not necessarily a bug, but someone asking for support

Comments

@Firstyear
Copy link
Contributor

cargo test miri fails, it appears to be an issue with no creating the directories correctly in tmpdir during the build process? I've put as much detail as I have below. Thanks!

RUST_BACKTRACE=1 cargo miri test 
    Finished release [optimized] target(s) in 0.05s
    Finished release [optimized] target(s) in 0.04s
error: intermittent IO error while iterating directory `/var/folders/5q/5nsgtt256133142xnr6f_lw80000gp/T/xargo.VmAuafc8TFD0/target/x86_64-apple-darwin/release/deps`
caused by: IO error for operation on /var/folders/5q/5nsgtt256133142xnr6f_lw80000gp/T/xargo.VmAuafc8TFD0/target/x86_64-apple-darwin/release/deps: No such file or directory (os error 2)
caused by: No such file or directory (os error 2)
   0: backtrace::backtrace::trace
   1: backtrace::capture::Backtrace::new
   2: error_chain::make_backtrace
   3: <core::result::Result<T,E> as xargo::errors::ResultExt<T,E>>::chain_err
   4: xargo::util::cp_r
   5: xargo::sysroot::build
   6: xargo::sysroot::update
   7: xargo::main_inner
   8: std::rt::lang_start::{{closure}}
   9: std::rt::lang_start_internal
  10: main

fatal error: Failed to run xargo

ls -al /var/folders/5q/5nsgtt256133142xnr6f_lw80000gp/T/
total 8
drwx------@ 13 william  staff  416 31 Mar 19:57 .
drwxr-xr-x@  5 william  staff  160  2 Jan  2019 ..
drwx------   6 william  staff  192 31 Mar 19:06 .AddressBookLocks
drwx------   2 william  staff   64 31 Mar 19:28 .CalendarLocks
drwxr-xr-x   3 william  staff   96 31 Mar 19:55 .LINKS
drwx------@  2 william  staff   64 31 Mar 19:55 TemporaryItems
drwx------@  3 william  staff   96 31 Mar 19:55 com.apple.mail
drwx------@  3 william  staff   96 31 Mar 19:30 com.apple.nsurlsessiond
drwx------   3 william  staff   96 31 Mar 19:17 com.apple.useractivityd
drwx------@  2 william  staff   64 31 Mar 19:28 com.nextcloud.desktopclient.FinderSyncExt
drwxr-xr-x   3 william  staff   96 31 Mar 19:39 new
-rw-------   1 william  staff  329 31 Mar 19:44 xcrun_db

env 
TMPDIR=/var/folders/5q/5nsgtt256133142xnr6f_lw80000gp/T/

toolchain nightly-2020-03-29-x86_64-apple-darwin
@Firstyear
Copy link
Contributor Author

The problem may actually be in xargo, it appears that it works out dst in ./src/sysroot.rs, then attempts a cp_r from it's ./src/util.rs, but the issue is dst would only be /var/folders/5q/5nsgtt256133142xnr6f_lw80000gp/T/xargo.VmAuafc8TFD0, and the dst has target/x86_64-apple-darwin/release/deps appended. Possibly a mkdir -p or eqivalent before starting the cp_r would resolve, but I'd want someone who knows more about miri and xargo to comment.

@RalfJung
Copy link
Member

Yeah the error definitely looks like it is on the xargo side.

I have no idea what an "intermittent IO error" is... does this happen each time, or only sometimes? If it happens each time, do you have any idea why your system would be affected when things work fine on macOS otherwise? (For example, Miri CI runs on macOS, and I never saw this failure.) And what is that strange 5nsgtt256133142xnr6f_lw80000gp directory name? (I'm on Linux, so macOS is all strange to me anyway.)

@RalfJung RalfJung added A-cargo Area: affects the cargo wrapper (cargo miri) A-mac Area: Affects only macOS targets C-bug Category: This is a bug. labels Mar 31, 2020
@Firstyear
Copy link
Contributor Author

Yeah the error definitely looks like it is on the xargo side.

I have no idea what an "intermittent IO error" is... does this happen each time, or only sometimes? If it happens each time, do you have any idea why your system would be affected when things work fine on macOS otherwise?

No idea - I have a clean install, and I had only just installed the miri + nightly toolchain. :( It's 100% reproducable for me. About the only thing I can think is that I'm on 10.15.4 so very latest macos.

(For example, Miri CI runs on macOS, and I never saw this failure.) And what is that strange 5nsgtt256133142xnr6f_lw80000gp directory name? (I'm on Linux, so macOS is all strange to me anyway.)

TMPDIR is randomised on macos for security, so mine is for this session TMPDIR=/var/folders/5q/5nsgtt256133142xnr6f_lw80000gp/T/, but could be different next boot etc. So this base location is fine, it's the behaviour below that which is the problem where the cp_r looks like it doesn't do a "mkdir -p" of the destination?

@RalfJung
Copy link
Member

RalfJung commented Mar 31, 2020 •
edited
Loading

the cp_r looks like it doesn't do a "mkdir -p" of the destination?

It already does, see here.

It looks like for you, walkdir is already failing. That's really odd. And why does it list that temp dir as src? I would expect it to copy to the tempdir, not from it.

The issue here isn't that it is copying to a non-existing directory -- it is trying to copy from something non-existent...

@RalfJung
Copy link
Member

So looking at the code, the failing cp_r is likely this one. This looks like xargo is already done with the sysroot build but then failing to copy the files out of the build dir to where they are needed.

Strangely, your log output above does not show any indication that a build is actually happening! Is that the full output? Could you try cargo miri setup, to explicitly request just a sysroot build without also running anything?

@Firstyear
Copy link
Contributor Author

So I ran with > XARGO_KEEP_TEMP=1 RUST_BACKTRACE=1 cargo miri setup -v, and I can see the following:

 > RUST_BACKTRACE=1 cargo miri setup -v
    Finished release [optimized] target(s) in 0.05s
    Finished release [optimized] target(s) in 0.03s
error: intermittent IO error while iterating directory `/var/folders/5q/5nsgtt256133142xnr6f_lw80000gp/T/xargo.EIMRVuFqJnPQ/target/x86_64-apple-darwin/release/deps`
caused by: IO error for operation on /var/folders/5q/5nsgtt256133142xnr6f_lw80000gp/T/xargo.EIMRVuFqJnPQ/target/x86_64-apple-darwin/release/deps: No such file or directory (os error 2)
caused by: No such file or directory (os error 2)
   0: backtrace::backtrace::trace
   1: backtrace::capture::Backtrace::new
   2: error_chain::make_backtrace
   3: <core::result::Result<T,E> as xargo::errors::ResultExt<T,E>>::chain_err
   4: xargo::util::cp_r
   5: xargo::sysroot::build
   6: xargo::sysroot::update
   7: xargo::main_inner
   8: std::rt::lang_start::{{closure}}
   9: std::rt::lang_start_internal
  10: main

fatal error: Failed to run xargo

> find /var/folders/5q/5nsgtt256133142xnr6f_lw80000gp/T/xargo.urpEDAyFvDLM/
/var/folders/5q/5nsgtt256133142xnr6f_lw80000gp/T/xargo.urpEDAyFvDLM/
/var/folders/5q/5nsgtt256133142xnr6f_lw80000gp/T/xargo.urpEDAyFvDLM//Cargo.toml
/var/folders/5q/5nsgtt256133142xnr6f_lw80000gp/T/xargo.urpEDAyFvDLM//Cargo.lock
/var/folders/5q/5nsgtt256133142xnr6f_lw80000gp/T/xargo.urpEDAyFvDLM//src
/var/folders/5q/5nsgtt256133142xnr6f_lw80000gp/T/xargo.urpEDAyFvDLM//src/lib.rs

So it looks like whatever xargo is attempting to build, isn't being built at all, so that's why the target dir is missing.

Even as an initial suggestion, perhaps in ./src/bin/cargo-miri.rs, if miri is set to -v, then to not set -q to build in xargo (line 380 I think)? It might also be good if with -v that the command and env is displayed before it's run on line 390 too so that we can more easily reproduce the failure, or see environmental issues that could be part of the problem? Also worth pointing out, that the error message should be "failed to run xargo-check" not "failed to run xargo" at list 390/391?

So from here I wanted to see why xargo-check was failing an to reproduce that environment, so I used the following shell script:

> cat /tmp/xargo-check
#!/bin/sh

if [ $1 == "--version" ]; then
    echo "xargo 0.3.20" >&2
    echo "cargo 1.44.0-nightly (8a0d4d9c9 2020-03-24)" >&2
    exit 0
fi

echo $(pwd) > /tmp/xargo-check.log
echo $(env) >> /tmp/xargo-check.log
echo $@ >> /tmp/xargo-check.log

And ran XARGO_CHECK=/tmp/xargo-check cargo miri setup. I then used that to get the environment and location of the build, and the output is as follows:

~/Library/Caches/org.rust-lang.miri > XARGO_RUST_SRC=/Users/william/.rustup/toolchains/nightly-2020-03-29-x86_64-apple-darwin/lib/rustlib/src/rust/src RUSTFLAGS="-Zalways-encode-mir -Zmir-emit-retag -Zmir-opt-level=0 --cfg=miri -Cdebug-assertions=on" CARGO=/Users/william/.rustup/toolchains/nightly-2020-03-29-x86_64-apple-darwin/bin/cargo XARGO_HOME=/Users/william/Library/Caches/org.rust-lang.miri RUSTUP_TOOLCHAIN=nightly-2020-03-29-x86_64-apple-darwin RUSTUP_HOME=/Users/william/.rustup /Users/william/.cargo/bin/xargo-check build -v
+ "rustc" "--print" "sysroot"
+ RUSTFLAGS="-Zalways-encode-mir -Zmir-emit-retag -Zmir-opt-level=0 --cfg=miri -Cdebug-assertions=on --sysroot /Users/william/Library/Caches/org.rust-lang.miri/HOST -Z force-unstable-if-unmarked"
+ "/Users/william/.rustup/toolchains/nightly-2020-03-29-x86_64-apple-darwin/bin/cargo" "check" "--release" "--manifest-path" "/var/folders/5q/5nsgtt256133142xnr6f_lw80000gp/T/xargo.et6IDRgFQx3p/Cargo.toml" "--target" "x86_64-apple-darwin" "-v" "-p" "std"
       Fresh core v0.0.0 (/Users/william/.rustup/toolchains/nightly-2020-03-29-x86_64-apple-darwin/lib/rustlib/src/rust/src/libcore)
       Fresh cc v1.0.50
       Fresh autocfg v0.1.7
       Fresh rustc-std-workspace-core v1.99.0 (/Users/william/.rustup/toolchains/nightly-2020-03-29-x86_64-apple-darwin/lib/rustlib/src/rust/src/tools/rustc-std-workspace-core)
       Fresh compiler_builtins v0.1.25
       Fresh alloc v0.0.0 (/Users/william/.rustup/toolchains/nightly-2020-03-29-x86_64-apple-darwin/lib/rustlib/src/rust/src/liballoc)
       Fresh cfg-if v0.1.10
       Fresh rustc-demangle v0.1.16
       Fresh libc v0.2.66
       Fresh rustc-std-workspace-alloc v1.99.0 (/Users/william/.rustup/toolchains/nightly-2020-03-29-x86_64-apple-darwin/lib/rustlib/src/rust/src/tools/rustc-std-workspace-alloc)
       Fresh unwind v0.0.0 (/Users/william/.rustup/toolchains/nightly-2020-03-29-x86_64-apple-darwin/lib/rustlib/src/rust/src/libunwind)
       Fresh backtrace-sys v0.1.35
       Fresh panic_abort v0.0.0 (/Users/william/.rustup/toolchains/nightly-2020-03-29-x86_64-apple-darwin/lib/rustlib/src/rust/src/libpanic_abort)
       Fresh backtrace v0.3.46
       Fresh panic_unwind v0.0.0 (/Users/william/.rustup/toolchains/nightly-2020-03-29-x86_64-apple-darwin/lib/rustlib/src/rust/src/libpanic_unwind)
       Fresh hashbrown v0.6.2
       Fresh std v0.0.0 (/Users/william/.rustup/toolchains/nightly-2020-03-29-x86_64-apple-darwin/lib/rustlib/src/rust/src/libstd)
    Finished release [optimized] target(s) in 0.05s
+ RUSTFLAGS="-Zalways-encode-mir -Zmir-emit-retag -Zmir-opt-level=0 --cfg=miri -Cdebug-assertions=on --sysroot /Users/william/Library/Caches/org.rust-lang.miri/HOST -Z force-unstable-if-unmarked"
+ "/Users/william/.rustup/toolchains/nightly-2020-03-29-x86_64-apple-darwin/bin/cargo" "check" "--release" "--manifest-path" "/var/folders/5q/5nsgtt256133142xnr6f_lw80000gp/T/xargo.et6IDRgFQx3p/Cargo.toml" "--target" "x86_64-apple-darwin" "-v" "-p" "test"
       Fresh core v0.0.0 (/Users/william/.rustup/toolchains/nightly-2020-03-29-x86_64-apple-darwin/lib/rustlib/src/rust/src/libcore)
       Fresh cc v1.0.50
       Fresh autocfg v0.1.7
       Fresh rustc-std-workspace-core v1.99.0 (/Users/william/.rustup/toolchains/nightly-2020-03-29-x86_64-apple-darwin/lib/rustlib/src/rust/src/tools/rustc-std-workspace-core)
       Fresh compiler_builtins v0.1.25
       Fresh libc v0.2.66
       Fresh cfg-if v0.1.10
       Fresh alloc v0.0.0 (/Users/william/.rustup/toolchains/nightly-2020-03-29-x86_64-apple-darwin/lib/rustlib/src/rust/src/liballoc)
       Fresh rustc-demangle v0.1.16
       Fresh backtrace-sys v0.1.35
       Fresh unwind v0.0.0 (/Users/william/.rustup/toolchains/nightly-2020-03-29-x86_64-apple-darwin/lib/rustlib/src/rust/src/libunwind)
       Fresh panic_abort v0.0.0 (/Users/william/.rustup/toolchains/nightly-2020-03-29-x86_64-apple-darwin/lib/rustlib/src/rust/src/libpanic_abort)
       Fresh rustc-std-workspace-alloc v1.99.0 (/Users/william/.rustup/toolchains/nightly-2020-03-29-x86_64-apple-darwin/lib/rustlib/src/rust/src/tools/rustc-std-workspace-alloc)
       Fresh backtrace v0.3.46
       Fresh panic_unwind v0.0.0 (/Users/william/.rustup/toolchains/nightly-2020-03-29-x86_64-apple-darwin/lib/rustlib/src/rust/src/libpanic_unwind)
       Fresh hashbrown v0.6.2
       Fresh std v0.0.0 (/Users/william/.rustup/toolchains/nightly-2020-03-29-x86_64-apple-darwin/lib/rustlib/src/rust/src/libstd)
       Fresh rustc-std-workspace-std v1.99.0 (/Users/william/.rustup/toolchains/nightly-2020-03-29-x86_64-apple-darwin/lib/rustlib/src/rust/src/tools/rustc-std-workspace-std)
       Fresh term v0.0.0 (/Users/william/.rustup/toolchains/nightly-2020-03-29-x86_64-apple-darwin/lib/rustlib/src/rust/src/libterm)
       Fresh proc_macro v0.0.0 (/Users/william/.rustup/toolchains/nightly-2020-03-29-x86_64-apple-darwin/lib/rustlib/src/rust/src/libproc_macro)
       Fresh unicode-width v0.1.6
       Fresh getopts v0.2.21
       Fresh test v0.0.0 (/Users/william/.rustup/toolchains/nightly-2020-03-29-x86_64-apple-darwin/lib/rustlib/src/rust/src/libtest)
    Finished release [optimized] target(s) in 0.03s
error: intermittent IO error while iterating directory `/var/folders/5q/5nsgtt256133142xnr6f_lw80000gp/T/xargo.et6IDRgFQx3p/target/x86_64-apple-darwin/release/deps`
caused by: IO error for operation on /var/folders/5q/5nsgtt256133142xnr6f_lw80000gp/T/xargo.et6IDRgFQx3p/target/x86_64-apple-darwin/release/deps: No such file or directory (os error 2)
caused by: No such file or directory (os error 2)
note: run with `RUST_BACKTRACE=1` for a backtrace

Both of the executed commands are "check" commands, not "build" commands, which is quite likely why the xargo command is failing to copy artifacts - there are none, we only did a check!

Is this possibly a bug in xargo at this point?

> xargo -V
xargo 0.3.20
cargo 1.40.0 (bc8e4c8be 2019-11-22)

@Firstyear
Copy link
Contributor Author

You know what I realised it might be - I build into a ram disk, configured in my .cargo/config:

 > cat ~/.cargo/config
[build]
target-dir = "/Volumes/ramdisk/rs"

So I wonder if that's what's the problem, that target/ is landing outside of the project location because xargo/miri don't read the .cargo/config target-dir parameter perhaps?

@RalfJung
Copy link
Member

RalfJung commented Apr 1, 2020 •
edited
Loading

Ah yes, that would definitely be a problem. :D

Could you open a xargo issue that xargo does not work when target-dir is set? I doubt an absolute target-dir can work for xargo, we kind of require a clean dir to know what it is that we have to copy (everything in the target dir) -- and because otherwise cargo thinks it can be smart and skip the build of stuff that is already built, which might be why we see no build for you.

@RalfJung RalfJung added C-support Category: Not necessarily a bug, but someone asking for support and removed A-mac Area: Affects only macOS targets C-bug Category: This is a bug. labels Apr 1, 2020
@RalfJung
Copy link
Member

RalfJung commented Apr 1, 2020

Both of the executed commands are "check" commands, not "build" commands, which is quite likely why the xargo command is failing to copy artifacts - there are none, we only did a check!

Checking also creates artifacts (it creates files with crate metadata and MIR). This is needed so that other crates depending on this one can be checked.

Checking is what we do on all platforms for a few weeks already and it works fine on CI and for plenty of users (otherwise we would have heard about it ;). In fact, this was a crucial step towards #1198. So checking is not the problem.

@Firstyear Firstyear mentioned this issue Apr 1, 2020
Closed
@Firstyear
Copy link
Contributor Author

Done, xargo issue opened! Thanks for your patience! japaric/xargo#286

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
A-cargo Area: affects the cargo wrapper (cargo miri) C-support Category: Not necessarily a bug, but someone asking for support
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Firstyear @RalfJung