std::process::Command's current_dir behaves differently on Unix and Windows, with respect to relative exe paths

[oconnor663]

On Unix, relative paths to the exe are interpreted after cd'ing to the current_dir, but on Windows they're interpreted before. Here's an example test (1 test file, and 1 little binary that the test needs to build) that demonstrates this if you run cargo test on the two different platforms:

src/lib.rs

#[test]
fn execute_binary_from_another_dir() {
    // Run `cargo build` to make sure the associated `foo` binary gets built.
    std::process::Command::new("cargo").arg("build").status().unwrap();

    // Shell out to `foo`, just to confirm that this works fine.
    std::process::Command::new("target/debug/foo")
        .status()
        .unwrap();

    // Now try to shell out to `foo` again while also setting the current_dir.
    // This fails on Unix, because the path to the binary is resolved *after*
    // changing directories, but it works on Windows.
    std::process::Command::new("target/debug/foo")
        .current_dir("..")
        .status()
        .unwrap();
}

src/bin/foo.rs

fn main() {
    println!("foo!");
}

I ran into a similar difference when I tried this with Python's subprocess.run(..., cwd=...). It looks like it's an artifact of Windows supporting an lpCurrentDirectory arg natively, while Unix has to fork and then chdir in the child before it calls exec. I prefer the semantics that we're getting on Windows right now*, but this issue is mainly that they're not the same.

* My main reason for preferring the current-Windows-style "current_dir does not affect the exe location" semantics: Emulating the current-Unix-style only requires a Path::join, which always works. But going the other way requires a Path::canonicalize, which can fail.

[alexcrichton]

Interesting! I believe your diagnosis is spot on as well.

A possible fix to this would be to always pass an absolute path on Unix but that'd involve probing which would inevitably go wrong somewhere... I agree though that the Windows semantics make more sense to me here.

[oconnor663]

@alexcrichton would it be possible to exec the new program with fexecve instead of execvp? I think would let us open the executable file before we change directories, and then execute it afterwards from its file descriptor. But it doesn't look like that's exposed in libc yet, and I don't know how portable it is?

[alexcrichton]

mind blown

Whoa I had no idea that API even existed! The portability does worry me though, especially this clause from the man page:

On Linux, fexecve() is implemented using the proc(5) file system, so /proc needs to be mounted and available at the time of the call.

We've had instances before of Rust binaries running without /proc and random things in glibc break (like thread info stuff). Putting this on Command::new may be a bit hindering :(

[oconnor663]

duct is currently working around this by doing a canonicalize under the covers. My reasoning here is that errors are acceptable, because in the situations that trigger this (relative program name, current_dir is in use) it's pretty much guaranteed that the invocation would fail otherwise anyway, unless the caller was actively prepared for the Unix semantics. But it's easier to say that as a higher level library, than it is as the stdlib.

[rzhikharevich]

I've just ran into this problem (specifically, the Unix behavior being rather unintuitive and undocumented).

@alexcrichton could you please describe what's wrong with making an absolute path with canonicalize before forking?

[alexcrichton]

@rzhikharevich I don't know of anything that could concretely go wrong right now, I'm just assuming that because this is low-level code used by tons of applications that the solution is bound to not be as simple as "just use canonicalize". In implementing a fix for this we basically just need to proceed with caution.

[oconnor663]

I'll note that I've seen canonicalize fail before on a VirtualBox shared drive, because the filesystem itself didn't support it. In that case I think a fallback of manually traversing .. entries would actually have worked, but I wouldn't be surprised if that created other issues. Just spitballing, it's possible that a manual traversal would give a different answer than canonicalize would have -- since I think the filesystem is allowed to implement canonicalize however it likes -- which could then lead to really weird behavior if canonicalize was failing say 50% of the time.

Even though it's not the semantics I prefer, doing a Path::join on Windows so that we always get the Unix behavior might be simpler. And in the defense of that behavior, relative paths passed to the child process as command line arguments will of course interact with current_dir, so having the binary path interact too isn't totally inconsistent.

But either way, this would probably constitute a breaking change at this point. Or at least, we'd have to be super careful to check the ecosystem and see who's depending on these details. I might go ahead and open a PR to document the current behavior at least.

[retep998]

We should never be using canonicalize on Windows unless we want to resolve symbolic links. If we just want to make a relative path absolute we should be using GetFullPathNameW.

[oconnor663]

@alexcrichton @retep998 @rzhikharevich I just added a docs PR above. Please take a look if you have time.

[benesch]

Just stumbled across this myself. Or rather ran rather headfirst into it. The latest macOS makes this situation even worse, in that an invocation like

Command::new("./script").current_dir("some_dir").status().unwrap()

will both a) successfully spawn the process some_dir/script (assuming said file exists) and b) panic with an ENOENT error. This actually might be a Rust bug; it's not clear to me whether the ENOENT is something macOS's posix_spawn function is incorrectly generating or whether Rust is doing something that generates an ENOENT after calling posix_spawn.

What are our options here? Is there any way we can disallow current_dir when the command is relative? The current behavior is a massive footgun, IMO.

[m-ou-se]

This needs a proposal for moving forward, so we can provide feedback on the proposed solution.

@ChrisDenton Since you're working on the Windows-specific code around environment variables and Command recently, I'd like to hear your input on it, if you have an opinion on this issue.

[ChrisDenton]

Hm, I don't feel too strongly on this but I think resolving using the parent's current directory is more intuitive to me. In Windows the executable must be found before a new process is created which is unlike a fork-exec model that sets up the environment before locating the executable. Perhaps this accounts for my personal expectations.

That said, using the child's path should not be too hard to implement on the Windows side but if that were to happen I think it'd be great if the documentation for Command::new made this clear. It's easy to miss if it's only documented in Command::current_dir and I do think this would surprising behaviour.

Of course if *nix users would be happy with using the parent's current directory then I'd be happy too. Or if neither can be changed at this point then that's frustrating but understandable. In the future there could perhaps be an API to change how relative paths and plain file names are resolved but that's probably drifting off topic.

tl;dr I'd vote for using the parent's current directory but I'd not object if the decision goes another way.

[oconnor663]

I also prefer the Windows behavior here, though I'm not a Windows user myself. For example, imagine that both ./run.sh and ./foo/run.sh exist. Then we could write this:

let mut my_command = Command::new("./run.sh");
my_command.status().unwrap();  // the exe in this directory
my_command.current_dir("foo");
my_command.status().unwrap();  // the same exe on Windows, or the other one on Unix

I think "the same exe" is much less surprising, given that it's the same Command object and that the command path inside is otherwise immutable.

As far as what we can do about it, gosh I don't know. Even if we were willing to break some existing code on Unix, is there an implementation we'd be comfortable with that would give us Windows-like behavior? Calling realpath or similar introduces new error cases, which could conceivably break programs that aren't using relative paths at all. I don't know the standard filesystem APIs well enough to know what all our options are here.