Offline mode

[matklad]

Implementation PR: #4770
Original issue: #4686

Summary:

Running Cargo with -Z offline flag changes the resolution process such that it only considered crates, already downloaded locally.

Steps:

• currently, there's no way to populate local crate storage, see Add an "airplane" flag to Cargo, altering it's dependency resolution behavior #4686 for details

• write precise documentation about effects and intended use-case for this flag. See https://github.com/rust-lang/cargo/blob/5845464cf92a348f95dc6b43133c017c37fcb684/src/doc/src/reference/unstable.md#offline-mode for current docs

Stabilization TODO:

• change -Z offline unstable cli option to a stable one, which works the same way as --frozen and --locked flags do.
• -Z offline actively breaks building crate offline #6014

[matklad]

cc @Eh2406

[Eh2406]

Thanks for the write up. I think "no way to populate local crate storage" is overstated. There are pretty good, approximate, ways to do it like having used that dep in any previous project. Nead an ecosystem, build stdx, or equivalent and you have most of it. And complete, manually ways to do it, download all the crates and unpack them into carcos cache directory. It just takes a lot of disk space ~12 GB. Both of these the community can develop and improve once people are using offline. What is missing is a way to store a local compressed copy of crates and only uncompressed them as cargo needs them. That seems like it needs a lot of design work and could be an add on feature.

Given that, I think we should work to stabilize the existing offline, and come back to populateting local crate storage.

[alexcrichton]

One TODO item I might add as well is to audit the error messages and make sure they're understandable. For example if resolution fails due to the inability to update the registry then the error message should state that -Z offline was passed and removing it may allow the resolution to be fixed (if there's a missing version or something like that)

Otherwise I do personally still feel that population of the local storage is somewhat important. I see this as a sort of "what do I do before I get on a plane?" question. Or similarly what does Crater do before building all its crates? I think we'd want it along the lines of cargo cache stdx or something like that (aka just download a list of crates and all their transitive dependencies from crates.io) but I think we'd want to build that out in some command (the implementation shouldn't be too hard) before stabilizing.

[Eh2406]

An audit of the error messages is a good idea. I think the "-Z offline may be your problem" is in already. But the opposite would be good as well. "errore: no internet access; you can try -Z offline to try with what you have cashed"

I think that an initial version of a cargo cache can be an out of tree cargo-subcommand. That would allow the community to experiment in the design space, before we have to commit to an implementation.

[alexcrichton]

But the opposite would be good as well

An excellent idea!

I think that an initial version of a cargo cache can be an out of tree cargo-subcommand

I'd be totally on board with that

[luser]

I'm sure I mentioned this in a previous issue but I've used cargo-cacher for offline work before. It has a few options but I usually just run it with --all to fetch everything it can from crates.io. That always feels like overkill to me, though. Ideally I think I'd want some options like "fetch these specific crates and all their dependencies", "fetch the top N crates and all their dependencies", "fetch the newest version of every crate and all their dependencies".

[ehuss]

Is it intentional that offline mode will fail if an optional dependency is missing? It seems like it shouldn't because in online mode it doesn't fetch anything (AFAIK).

Example:

[package]
name = "off"
version = "0.1.0"

[dependencies]
# A very old version you are unlikely to have in your cache.
bitflags = {version="=0.1.0", optional=true}

> cargo +nightly build -Z offline
error: no matching version `= 0.1.0` found for package `bitflags`
location searched: registry `https://github.com/rust-lang/crates.io-index`
versions found: 1.0.3, 1.0.2, 1.0.1, ...
required by package `off v0.1.0 (file:///Users/eric/Proj/rust/cargo/scratch/off)`
As a reminder, you're using offline mode (-Z offline) which can sometimes cause surprising 
resolution failures, if this error is too confusing you may with to retry without the offline flag.

[alexcrichton]

@ehuss currently, yes, unfortunately. That's because lockfile generation requires the assumption that all possible features are enabled. Generating a lockfile without optional features is covered by #5133

[Jasper-Bekkers]

A similar situation seems to happen with target specific dependencies. Something with this in the Cargo.toml will fail to build on windows with -Z offline:

[target.x86_64-unknown-linux-gnu.dependencies]
x11-dl = "~2.14"

Giving

error: no matching package named `x11-dl` found

[alexcrichton]

@Jasper-Bekkers is that the full error message? Does Cargo print any other information?

[Jasper-Bekkers]

It's pretty much the full error message.

>cargo build -Z offline
error: no matching package named `x11-dl` found
location searched: registry `https://github.com/rust-lang/crates.io-index`
required by package `minifb v0.10.7`
    ... which is depended on by `voxelize-rs v0.1.0 (file:///...)`
As a reminder, you're using offline mode (-Z offline) which can sometimes cause surprising resolution failures, if this error is too confusing you may with to retry without the offline flag.

[alexcrichton]

@Jasper-Bekkers ah ok, did that not indicate enough that -Z offline was likely at fault?

[Jasper-Bekkers]

-Z offline is indeed at fault; it's evaluating and matching Linux dependencies on a Windows machine while it shouldn't in similar vein to the broken optional dependencies. I think both should be made to work for this to be a useful workflow (eg. the airplane use case).

I figured I'd bump this but if a new ticket is needed then I can file a new one.

[alexcrichton]

@Jasper-Bekkers oh that issue is different from this one and is covered by #5133

[Jasper-Bekkers]

I don't think it does: #5133 (comment)

[alexcrichton]

That's an implementation bug, not a reflection of the feature-to-be-stabilized

[jrvanwhy]

Can we have this as a .cargo/config setting as well?

Use case: a project that wants to vendor all their dependencies and compile without making external network requests (e.g. for technical reasons or auditing purposes). Changing cargo's default behavior repository-wide would make maintaining the build system much easier, as regressions would become apparent more quickly.

[ehuss]

I do personally still feel that population of the local storage is somewhat important.

I have published a subcommand to download popular crates: https://crates.io/crates/cargo-prefetch
It is pretty basic at this point (it relies on Cargo to actually download things). I figure it can be changed as needed once people have some experience with it, and gauge how useful it is.

[nrc]

What is the status of offline mode? Other than stabilising --offline what remains to be done? Are there any blocking bugs?

[alexcrichton]

I believe the last major blocker was something like @ehuss's cargo prefetch, and now I think @ehuss you're putting the finishing touches on the error messages and documentation and we should be good to go?

[ehuss]

Yea, once we decide the fate of #6871 I was going to make a stabilization report and fcp request.

[ehuss]

This is a proposal to stabilize the offline feature.

@rfcbot fcp merge

Stabilization Target: 1.36 — Release date July 4 2019
Original Issue: #4686

What is being stabilized

The offline feature adds a flag to prevent Cargo from accessing the network, and to perform dependency resolution if the required dependencies are already downloaded.

I propose making the following changes when stabilizing:

• Change -Zoffline to a global flag --offline.
• Add a net.offline config value for an alternate way to set offline mode (also CARGO_NET_OFFLINE=true environment variable).

The offline mode behavior is fairly straightforward. An overview of when running in offline mode:

• The resolver will avoid updating the index, and if the required packages are already downloaded, then it will proceed as normal. If the required packages are not available, then it will give an error with a hint to run without offline mode.
• If the index has never been downloaded, it will give an error suggesting running without offline.
• Commands that must use the network, like cargo publish, provide a straightforward error message that it cannot be used in offline mode.

cargo fetch may be used to download an individual package's dependencies before going offline. I have published an experimental command cargo-prefetch to download popular dependencies.

[rfcbot]

Team member @ehuss has proposed to merge this. The next step is review by the rest of the tagged team members:

• @Eh2406
• @alexcrichton
• @ehuss
• @joshtriplett
• @nrc
• @withoutboats

No concerns currently listed.

Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

See this document for info about what commands tagged team members can give me.

[rfcbot]

🔔 This is now entering its final comment period, as per the review above. 🔔

[rfcbot]

The final comment period, with a disposition to merge, as per the review above, is now complete.

As the automated representative of the governance process, I would like to thank the author for their work and everyone else who contributed.

The RFC will be merged soon.

[anb0s]

I'm using "cargo 1.35.0 (6f3e9c3 2019-04-04)" and tried to get "cargo build" working with pre-cached crates, but had no luck. I've tried "cargo build --offline" and also CARGO_NET_OFFLINE=true environment variable. Any hints how to use it? Thanks!

[anb0s]

Ok i've just updated to beta "cargo 1.36.0-beta (c4fcfb7 2019-05-15)" and the option --offline is now available. Thanks and sorry.