Support platform-defined standard directories #5183

soc · 2018-03-15T00:50:24Z

This change stops cargo from violating the operating system rules
regarding the placement of config, cache, ... directories on Linux,
macOS and Windows.

Existing directories and overrides are retained.

The precedence is as follows:

use the CARGO_HOME environment variable if it exists (legacy)
use CARGO_CACHE_DIR, CARGO_CONFIG_DIR etc. env vars if they exist
use the ~/.cargo directory if it exists (legacy)
follow operating system standards

A new cargo command, dirs, is added, which can provide path
information to other command line tools.

Fixes:
#1734
#1976
rust-lang/rust#12725

Addresses:
rust-lang/rfcs#1615
#148,
#3981

rust-highfive · 2018-03-15T00:50:35Z

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @alexcrichton (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

Please see the contribution instructions for more information.

soc · 2018-03-15T00:51:38Z

Hi everyone, I would love to get some feedback on this (and I couldn't reach anyone at #cargo).
There are probably still things that are wrong and need to be adapted, but I'd like to get some feedback early.

soc · 2018-03-15T03:06:05Z

New cargo dirs command

Examples:

with existing .cargo directory

$ target/debug/cargo dirs
CARGO_CACHE_DIR:  "/home/soc/.cargo"
CARGO_CONFIG_DIR: "/home/soc/.cargo"
CARGO_DATA_DIR:   "/home/soc/.cargo"
CARGO_BIN_DIR:    "/home/soc/.cargo/bin"

without existing .cargo directory

$ target/debug/cargo dirs
CARGO_CACHE_DIR:  "/home/soc/.cache/cargo"
CARGO_CONFIG_DIR: "/home/soc/.config/cargo"
CARGO_DATA_DIR:   "/home/soc/.local/share/cargo"
CARGO_BIN_DIR:    "/home/soc/.local/bin/"

soc · 2018-03-15T03:31:53Z

src/cargo/util/config.rs

+    // This is written in the most straight-forward way possible, because it is
+    // hard as-is to understand all the different options, without trying to
+    // save lines of code.
+    pub fn cargo_dirs() -> CargoDirs {


Should Config::cargo_dirs be renamed/moved to CargoDirs::new?

Yeah, new would looks slightly better I think, though current option is OK as well!

bors · 2018-03-15T05:22:25Z

☔ The latest upstream changes (presumably #5176) made this pull request unmergeable. Please resolve the merge conflicts.

aturon · 2018-03-16T04:34:46Z

cc @nrc

matklad

Overall, this looks great to me @soc!

One thing I am really worried about is how are we going to test this =/

we want to test it across at least mac/windows/linux (and probably on windows, there's also msvc/mingw/sigwin/linux subsystem axis?)
we want to test different fallback scenarios
we want to test this in conjunction with rustup

All this together implies to me that plain #[test] tests ain't gonna work here at all :(

I am thinking about a really heavy weight solution, like preparing docker images for different initial state of the machines, and then writing tests as bash scripts, which install rustup, create cargo project, etc. But, one does not simply create a docker image for windows I guess 🤷‍♂️ ?

It's also interesting that this PR actually does two things:

it refactors Cargo to support CargoDirs instead of monolithic CARGO_HOME.
it changes default locations for stuff, using directories and environment variables.

I wonder if it makes sense to split this over two pull request, and implement a refactoring first, while preserving current behavior fully. That way, we can separately check that the refactoring does not introduce regressions by itself, and then maximally concentrate on the fallback bits.

@rust-lang/cargo

matklad · 2018-03-17T09:32:52Z

src/cargo/ops/cargo_install.rs

@@ -122,7 +143,7 @@ pub fn install(
    if installed_anything {
        // Print a warning that if this directory isn't in PATH that they won't be
        // able to run these commands.
-        let dst = metadata(opts.config, &root)?.parent().join("bin");
+        let dst = metadata(opts.config, &Filesystem::new(root.config_dir))?.parent().join("bin");


This should be root.bin_dir I guess?

matklad · 2018-03-17T09:36:39Z

src/cargo/util/config.rs

+    // This is written in the most straight-forward way possible, because it is
+    // hard as-is to understand all the different options, without trying to
+    // save lines of code.
+    pub fn cargo_dirs() -> CargoDirs {


Yeah, new would looks slightly better I think, though current option is OK as well!

matklad · 2018-03-17T09:45:21Z

src/cargo/util/config.rs

+        let home_dir = ::home::home_dir().ok_or_else(|| {
+            format_err!("Cargo couldn't find your home directory. \
+                         This probably means that $HOME was not set.")
+        }).unwrap();


I think unwraps may panic in some obscure, yet real world scenario, so we really need proper error handlng. Changing the return type to CargoResult<CargoDirs> and replacing .unwraps with ? should do the trick I think?

As for an example of weird scenario, there were bug in Cargo about ::std::env::current_exe call failing, because Cargo was executed in chroot without procfs :)

matklad · 2018-03-17T09:46:43Z

src/cargo/util/config.rs

+        let mut cache_dir = cargo_dirs.cache_dir().to_path_buf();
+        let mut config_dir = cargo_dirs.config_dir().to_path_buf();
+        let mut data_dir = cargo_dirs.data_dir().to_path_buf();
+        // fixme: executable_dir only available on Linux, use data_dir on macOS and Windows?


This should be figured out in the RFC thread perhaps?

matklad · 2018-03-17T09:49:41Z

src/cargo/util/config.rs

+
+        // 3. .cargo exists
+        let legacy_cargo_dir = home_dir.join(".cargo");
+        if cargo_home_env.is_none() && legacy_cargo_dir.exists() {


This should be strictly a fall-back perhaps? That is, if all variables are not set, we set all dirs from .cargo, in contrast with the current per-dir approach?

matklad · 2018-03-17T09:51:14Z

src/cargo/util/config.rs

-    for current in paths::ancestors(pwd) {
-        let possible = current.join(".cargo").join("config");
+    for current in paths::ancestors(&dirs.current_dir) {
+        let possible = current.join(".cargo").join("config"); // fixme: what to do about this?


Nothing to be done here? It's an explicit feture of Cargo that it looks for .cargo/config in call parent directories. This exists for per-project .cargo dirs. Or am I missing something here?

Yes, I think you are right. I just wanted to be extra careful by marking all changes where I wasn't absolutely sure.

soc · 2018-03-17T12:48:46Z

@matklad Yes, testing is also a concern for me.

I think as a first step it would really help to have a list of different configurations and scenarios, so it is at least possible to test everything in an organized fashion, even if it is done manually.

I'm not sure whether splitting things into two commits makes sense, I feel that having an additional transitory state would have some benefits, but would also add overhead that would outweigh the benefits.

Especially because a similar PR needs to be done for rustup. With changes in one commit we would only have to test 4 different setups, {cargo: pre-change, post-change} * {rustup pre-change, post-change}.

With an additional state in the middle, this would balloon up to 9 different setups, for which all configurations and scenarios need to be tested against.

matklad · 2018-04-09T20:29:09Z

@soc have you managed to prepare a similar PR for rustup as well? I think updating rusupt would be a next step here, because we really do want to land changes to rustup and cargo simultaneously :)

soc · 2018-04-10T20:38:51Z

@matklad Not yet, but here is my plan:

I could really need some review of the current code to get a better understanding if this is the way people want to go (especially in regard to Path vs. Filesystem).
I would be thankful if someone has some time to look into the remaining test failures with me. I can make tests pass, but sometimes I'm not sure my changes to the tests are testing what was originally intended.
Implement changes in rustup.
Write some documentation that explains the changes.
Fix the existing documentation.
- Fix the official documentation.
- Fix third-party tutorials, blogs, StackOverflow, etc.
- Ping book authors.

matklad · 2018-04-11T10:12:04Z

src/cargo/ops/cargo_install.rs

@@ -235,9 +256,9 @@ fn install_one(
    // We have to check this again afterwards, but may as well avoid building
    // anything if we're gonna throw it away anyway.
    {
-        let metadata = metadata(config, root)?;
+        let metadata = metadata(config, &Filesystem::new(dirs.config_dir.clone()))?;


It feels like metadata could perhaps be a method of CargoInstallDirs?

matklad · 2018-04-11T10:14:28Z

src/cargo/util/config.rs

+    pub cache_dir: Filesystem,
+    pub config_dir: Filesystem,
+    pub data_dir: PathBuf,
+    pub bin_dir: PathBuf,


It would be great to add short docstrings here, to explain which directory stores which data.

matklad · 2018-04-11T10:18:03Z

src/cargo/util/config.rs

@@ -32,22 +33,119 @@ use util::Filesystem;

 use self::ConfigValue as CV;

+#[derive(Clone, Debug)]
+pub struct CargoDirs {
+    pub home_dir: Filesystem,


We don't actually use this field I think?

matklad · 2018-04-11T10:19:52Z

src/cargo/util/config.rs

+        let home_dir = ::home::home_dir().ok_or_else(|| {
+            format_err!("Cargo couldn't find your home directory. \
+                         This probably means that $HOME was not set.")
+        })?;


So looks like we don't use home_dir for anything except fallback, so let's move this as far down as possible, so that we don't actually fail if HOME is not set, but explicit directories are! We might want to test this behavior as well: working without home directory is great for reproducible and isolated builds.

matklad · 2018-04-11T10:22:44Z

src/cargo/util/config.rs

+        #[cfg(target_os = "macos")]
+        let _bin_dir = cargo_dirs.data_dir().parent().map(|p| p.join("bin"));
+        #[cfg(target_os = "windows")]
+        let _bin_dir = cargo_dirs.data_dir().parent().map(|p| p.join("bin"));


Let's extract this into a function

#[cfg(os = )] fn bin_dir(dirs: &ProjectDirs) -> Option<PathBuf>

matklad · 2018-04-11T10:29:52Z

src/cargo/util/config.rs

        } else if let Some(val) = self.get_path("build.target-dir")? {
-            let val = self.cwd.join(val.val);
+            let val = self.dirs.current_dir.join(val.val);


Let's use self.cwd() here for future-proofing.

matklad · 2018-04-11T10:36:33Z

src/cargo/util/config.rs

-        let home_path = self.home_path.clone().into_path_unlocked();
-        let credentials = home_path.join("credentials");
+        let config_path = self.dirs.config_dir.clone().into_path_unlocked();
+        let credentials = config_path.join("credentials");


Hm, and what is the "correct" place for credentials? config might be not the right place, because people sometimes publish it... I suggest raising this question on the RFC thread, if it wasn't raised already.

matklad · 2018-04-11T10:36:47Z

src/cargo/util/config.rs

@@ -638,7 +736,7 @@ impl Config {
                None => false,
            };
            let path = if maybe_relative {
-                self.cwd.join(tool_path)


matklad · 2018-04-11T10:46:50Z

tests/testsuite/login.rs

@@ -163,7 +163,7 @@ fn new_credentials_is_used_instead_old() {
        execs().with_status(0),
    );

-    let config = Config::new(Shell::new(), cargo_home(), cargo_home());
+    let config = Config::new(Shell::new(), CargoDirs::new().unwrap());


Here I think we want to point CargoDirs to CARGO_HOME inside test root.

One option is to provide another constructor for CargoDirs which accepts cargo_home: PathBuf. Another option is to modify the test such that it reads the config file directly. I am slightly in favor of the second approach.

matklad · 2018-04-11T11:05:02Z

tests/testsuite/cross_compile.rs

@@ -910,7 +910,7 @@ fn build_script_needed_for_host_and_target() {
    );
 }

-#[test]
+//#[test]
 fn build_deps_for_the_right_arch() {


This actually passes for me locally

alexcrichton · 2018-04-11T15:39:14Z

r? @matklad

alexcrichton · 2018-10-12T17:31:05Z

I'm going to close this because it's been stale and quiet for quite some time now unfortunately. The Cargo team is pretty tied up until after the 2018 edition, but I think we can perhaps look to help out integrating this and fixing remaining issues after the edition release.

flying-sheep · 2019-01-08T14:06:50Z

Hi! Rust 2018 is here, it’s a fresh new year! Time to get this rolling 😄

flying-sheep · 2019-01-08T14:08:14Z

Uh, how did I manage to unassign @matklad just by commenting‽

joshtriplett · 2019-01-09T21:09:10Z

src/cargo/util/config.rs

+              && cargo_cache_env.is_none()
+              && cargo_config_env.is_none()
+              && cargo_data_env.is_none()
+              && cargo_bin_env.is_none() {


This doesn't seem to handle the case where some but not all of the environment variables are set.

joshtriplett · 2019-01-09T21:15:47Z

@soc Let's see if we can get this revived.

I posted one comment for a corner case this doesn't seem to handle.

Could you please update this to fix the conflicts, and ensure that it still passes tests?

And could you please add tests for the various configuration cases (existing legacy configuration, existing XDG configuration, both, no existing configuration), to make sure they all work as expected?

FranklinYu · 2019-03-31T02:57:38Z

I don’t like the solution for macOS. See dirs-dev/directories-rs#47.

soc · 2019-03-31T12:20:59Z

@joshtriplett Sorry for the late reply.

I think my changes so far are flawed in the sense that all the existing logic should be retained as-is, otherwise it becomes really really hard to make sure the current behavior is the same without flipping the hypothetical switch to the new structure.

Sadly, at the moment I have little time for this (and the website shenanigans didn't help either).

I can offer some advice, though: As a first step, identify each an every place in cargo, rustup, etc. that uses the current structure and add an explicit branch with a check for the proposed -Z flag there.

Then test that all tools reach the right branch when the flag is set/not set.

Only at this point it makes sense to even start working on the new structure

RalfJung · 2019-06-18T09:24:11Z

src/cargo/util/config.rs

+                bin_dir    = legacy_cargo_dir.join("bin");
+            // 4. ... otherwise follow platform conventions
+            } else {
+                let cargo_dirs = ProjectDirs::from("", "", "Cargo");


FWIW, in Miri we will likely use ProjectDirs::from("org", "rust-lang", "miri") to get more descriptive names on platforms that commonly do that. Might be a good idea to do the same here?

alexcrichton · 2019-07-19T18:50:50Z

I'm going to close this for now because it's been languishing for some time, but if someone is willing to take this up again and resubmit it the Cargo team would be interested in finding a reviewer for it!

This commit is a continuation and adaptation of rust-lang#5183, which aimed to make cargo no longer reliant on the `$HOME/.cargo` directory in user's home's, and instead uses the `directories` crate to get platform-defined standard directories for data, caches, and configs. The priority of paths cargo will check is as follows: 1. Use `$CARGO_HOME`, if it is set 2. Use `$CARGO_CACHE_DIR`, `$CARGO_CONFIG_DIR`, etc, if they are set 3. If no environment variables are set, and `$HOME/.cargo` is present, use that 4. Finally, use the platform-default directory paths

soredake · 2020-04-18T08:47:06Z

Any progress on this?

luis-guimaraes-exoawk · 2022-07-11T10:17:55Z

Any updates?

Refs. - https://rust-lang.github.io/rustup/environment-variables.html - https://doc.rust-lang.org/cargo/reference/environment-variables.html See also: - https://wiki.archlinux.org/title/XDG_Base_Directory#Partial - rust-lang/rustup#247 - rust-lang/cargo#1734 - rust-lang/rfcs#1615 - rust-lang/cargo#5183 - rust-lang/cargo#148

LucasFA · 2024-02-13T15:10:56Z

For anyone interested, there is currently a pre-RFC in the forums:
https://internals.rust-lang.org/t/pre-rfc-split-cargo-home/19747

rust-highfive assigned alexcrichton Mar 15, 2018

soc force-pushed the topic/standard-directories branch from 6065027 to 390edbb Compare March 15, 2018 01:05

soc force-pushed the topic/standard-directories branch from 390edbb to c192c7f Compare March 15, 2018 03:09

soc commented Mar 15, 2018

View reviewed changes

soc force-pushed the topic/standard-directories branch from c192c7f to a290016 Compare March 15, 2018 03:34

soc force-pushed the topic/standard-directories branch from a290016 to 0b350ad Compare March 15, 2018 11:04

soc changed the title ~~[WIP] Add support for platform-defined standard directories~~ Add support for platform-defined standard directories Mar 17, 2018

matklad reviewed Mar 17, 2018

View reviewed changes

soc force-pushed the topic/standard-directories branch 4 times, most recently from cc80517 to 3aca0cf Compare March 18, 2018 16:10

Eh2406 mentioned this pull request Mar 28, 2018

Use OS-native config, cache home directories #1734

Open

matklad added the Rust-2018 label Mar 29, 2018

ehuss mentioned this pull request Mar 31, 2018

rfc: make cargo install extensible rust-lang/rfcs#2376

Closed

soc force-pushed the topic/standard-directories branch 2 times, most recently from ff5e99a to bef5ad1 Compare April 11, 2018 02:24

matklad reviewed Apr 11, 2018

View reviewed changes

soc force-pushed the topic/standard-directories branch from 750edc1 to 5d807d8 Compare April 11, 2018 14:51

reitermarkus mentioned this pull request Sep 4, 2018

rust/cargo: enable caching Homebrew/brew#4815

Merged

6 tasks

alexcrichton closed this Oct 12, 2018

flying-sheep unassigned matklad Jan 8, 2019

joshtriplett reviewed Jan 9, 2019

View reviewed changes

joshtriplett reopened this Jan 9, 2019

dwijnand removed the Rust-2018 label Feb 11, 2019

nrc assigned joshtriplett Mar 6, 2019

RalfJung reviewed Jun 18, 2019

View reviewed changes

alexcrichton closed this Jul 19, 2019

alexarchambault mentioned this pull request Jan 5, 2020

Unify cache directory across all OSes coursier/coursier#1483

Closed

spacekookie mentioned this pull request Apr 1, 2020

Use platform-defined directories for cargo state #8063

Closed

soc mentioned this pull request Mar 24, 2021

Create 0000_cargo_standard_directories.md spacekookie/rfcs#1

Open

machupicchubeta mentioned this pull request Oct 29, 2023

Move apps config and/or data directory under XDG Base Directory machupicchubeta/dotfiles#451

Merged

Support platform-defined standard directories #5183

Support platform-defined standard directories #5183

Conversation

soc commented Mar 15, 2018 • edited Loading

rust-highfive commented Mar 15, 2018

soc commented Mar 15, 2018

soc commented Mar 15, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bors commented Mar 15, 2018

aturon commented Mar 16, 2018

matklad left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

soc commented Mar 17, 2018 • edited Loading

matklad commented Apr 9, 2018

soc commented Apr 10, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexcrichton commented Apr 11, 2018

alexcrichton commented Oct 12, 2018

flying-sheep commented Jan 8, 2019

flying-sheep commented Jan 8, 2019

Choose a reason for hiding this comment

joshtriplett commented Jan 9, 2019 • edited Loading

FranklinYu commented Mar 31, 2019

soc commented Mar 31, 2019

Choose a reason for hiding this comment

alexcrichton commented Jul 19, 2019

soredake commented Apr 18, 2020

luis-guimaraes-exoawk commented Jul 11, 2022

LucasFA commented Feb 13, 2024

soc commented Mar 15, 2018 •

edited

Loading

soc commented Mar 15, 2018 •

edited

Loading

soc commented Mar 17, 2018 •

edited

Loading

soc commented Apr 10, 2018 •

edited

Loading

joshtriplett commented Jan 9, 2019 •

edited

Loading