Add support local mirrors of registries, take 2

[alexcrichton]

This series of commits culminates in first class support in Cargo for local mirrors of registries. This is implemented through a number of other more generic mechanisms, and extra support was added along the way. The highlights of this PR, however, are:

Source redirection

New .cargo/config keys have been added to enable replacing one source with another. This functionality is intended to be used for mirrors of the main registry or otherwise one to one source correspondences. The support looks like:

# in .cargo/config
[source.crates-io]
replace-with = 'my-awesome-registry'

[source.my-awesome-registry]
registry = 'https://github.com/my-awesome/registry-index'

This configuration means that instead of using crates-io (e.g. https://github.com/rust-lang/crates.io-index), Cargo will query the my-awesome-registry source instead (configured to a different index here). This alternate source must be the exact same as the crates.io index. Cargo assumes that replacement sources are exact 1:1 mirrors in this respect, and the following support is designed around that assumption.

When generating a lock file for crate using a replacement registry, the original registry will be encoded into the lock file. For example in the configuration above, all lock files will still mention crates.io as the registry that packages originated from. This semantically represents how crates.io is the source of truth for all crates, and this is upheld because all replacements have a 1:1 correspondance.

Overall, this means that no matter what replacement source you're working with, you can ship your lock file to anyone else and you'll all still have verifiably reproducible builds!

Adding sha256 checksums to the lock file

With the above support for custom registries, it's now possible for a project to be downloading crates from any number of sources. One of Cargo's core goals is reproducible builds, and with all these new sources of information it may be easy for a few situations to arise:

1. A local replacement of crates.io could be corrupt
2. A local replacement of crates.io could have made subtle changes to crates

In both of these cases, Cargo would today simply give non-reproducible builds. To help assuage these concerns, Cargo will now track the sha256 checksum of all crates from registries in the lock file. Whenever a Cargo.lock is generated from now on it will contain a [metadata] section which lists the sha256 checksum of all crates in the lock file (or <none> if the sha256 checksum isn't known).

Cargo already checks registry checksums against what's actually downloaded, and Cargo will now verify between iterations of the lock file that checksums remain the same as well. This means that if a local replacement registry is not in a 1:1 correspondance with crates.io, the lock file will prevent the build from progressing until the discrepancy is resolved.

Local Registries

In addition to the support above, there is now a new kind of source in Cargo, a "local registry", which is intended to be a subset of the crates.io ecosystem purposed for a local build for any particular project here or there. The way to enable this looks like:

# in .cargo/config
[source.crates-io]
replace-with = 'my-awesome-registry'

[source.my-awesome-registry]
local-registry = 'path/to/my/local/registry'

This local registry is expected to have two components:

1. A directory called index which matches the same structure as the crates.io index. The config.json file is not required here.
2. Inside the registry directory are any number of .crate files (downloaded from crates.io). Each crate file has the name <package>-<version>.crate.

This local registry must currently be managed manually, but I plan on publishing and maintaining a Cargo subcommand to manage a local registry. It will have options to do things like:

1. Sync a local registry with a Cargo.lock
2. Add a registry package to a local registry
3. Remove a package from a local registry

Directory registries

In addition to local registries, Cargo also supports a "directory source" like so

# in .cargo/config
[source.crates-io]
replace-with = 'my-awesome-registry'

[source.my-awesome-registry]
directory = 'path/to/some/sources'

A directory source is similar to a local registry above, except that all the crates are unpacked and visible as vendored source. This format is suitable for checking into source trees, like Gecko's.

Unlike local registries above we don't have a tarball to verify the crates.io checksum with, but each vendored dependency has metadata containing what it would have been. To further prevent modifications by accident, the metadata contains the checksum of each file which should prevent accidental local modifications and steer towards [replace] as the mechanism to edit dependencies if necessary.

What's all this for?

This is quite a bit of new features! What's all this meant to do? Some example scenarios that this is envisioned to solve are:

1. Supporting mirrors for crates.io in a first class fashion. Once we have the ability to spin up your own local registry, it should be easy to locally select a new mirror.
2. Supporting round-robin mirrors, this provides an easy vector for configuration of "instead of crates.io hit the first source in this list that works"
3. Build environments where network access is not an option. Preparing a local registry ahead-of-time (from a known good lock file) will be a vector to ensure that all Rust dependencies are locally available.
   • Note this is intended to include use cases like Debian and Gecko

What's next?

Even with the new goodies here, there's some more vectors through which this can be expanded:

• Support for running your own mirror of crates.io needs to be implemented to be "easy to do". There should for example be a cargo install foo available to have everything "Just Work".
• Replacing a source with a list of sources (attempted in round robin fashion) needs to be implemented
• Eventually this support will be extended to the Cargo.toml file itself. For example:
  • packages should be downloadable from multiple registries
  • replacement sources should be encodable into Cargo.toml (note that these replacements, unlike the ones above, would be encoded into Cargo.lock)
  • adding multiple mirrors to a Cargo.toml should be supported
• Implementing the subcommand above to manage local registries needs to happen (I will attend to this shortly)

[rust-highfive]

@alexcrichton: no appropriate reviewer found, use r? to override

[alexcrichton]

This is a rebasing of #2361 with directory support added in, I'll continue to follow up with more comments in a bit (hit the "make a PR" button a bit soon...)

[alexcrichton]

Ok,

cc @rust-lang/tools, a significant addition to Cargo with lots of room to grow as well. This is likely the start of Cargo supporting multiple registries. A next major work item here is to actually make it possible to bring your own mirror registry of crates.io online. That is, write the server which mirrors crates.io and otherwise ships crates to you.

Most of that shouldn't be too hard, but it'll be hard to get it nice and ergonomic the first time around I believe.

cc @luser, this is the support in Cargo necessary for vendoring dependencies in Gecko. The workflow we imagine is:

• When Gecko is checked out, it has a bunch of vendored Rust dependencies from crates.io
• When Gecko is configured, you have an option of developing Rust in "online mode" or "offline mode", the former hitting crates.io and the latter using the vendored sources. The switch is communicated to Cargo via a bit of .cargo/config configuration which is not checked into the source tree.
• Cargo will strive very hard to ensure that online mode and offline mode all compile the exact same thing so long as you have the same Cargo.lock. To that end Cargo will reject modifications to the vendored source code, instead suggesting the use of [replace] which is intended exactly for making modifications to a dependency quickly. I can go into more detail if you'd like, but it suffices to say that the workflow for modifying and testing out changes to dependencies should be quite smooth.
• Whenever a new dependency is added, someone with the cargo vendor subcommand will have to re-run it to ensure the "offline mode" build will continue to work. This has yet to be written, but will likely have syncing options and other various configuration (also shouldn't be too hard to write).

[cardoe]

So just so I'm clear, the local registry directory layout should look like:

.
└── local-registry
    ├── index
    │   └── git-clone-or-snapshot-crates.io-index
    ├── memchr-0.1.11.crate
    ├── rand-0.3.14.crate
    ├── rustfmt-0.5.0.crate
    └── semver-0.2.3.crate

And the plan is to nail this down as ABI so people can depend on this layout to remain constant?

[alexcrichton]

@cardoe indeed! Actually that reminds me that I forgot to write documentation on this in src/doc. I plan on documenting the format of local registries as well as directory sources so external tools can create and manage those workspaces.

I've previously written cargo local-registry for creating local registries, and I plan on updating cargo vendor to emit a directory source (this PR implements) once this is merged.

[luser]

This all sounds great. The directory source should meet our needs exactly.

Cargo will strive very hard to ensure that online mode and offline mode all compile the exact same thing so long as you have the same Cargo.lock. To that end Cargo will reject modifications to the vendored source code, instead suggesting the use of [replace] which is intended exactly for making modifications to a dependency quickly. I can go into more detail if you'd like, but it suffices to say that the workflow for modifying and testing out changes to dependencies should be quite smooth.

This seems pretty reasonable. We'll have to nail down (and document) the process for making changes to vendored libraries. I know you or @wycats mentioned a potential cargo fork workflow which would be great. I think this is one of those things that we're just going to have to learn as we go along, although I'm sure Servo has some great experience we can build off of.

[Ericson2314]

@alexcrichton I think I'm starting to understand this better :). If the section is called [source], perhaps any source including our existing git and path sources could be used as a mirror? Might that mean there would be no need for directory registries?

[aneeshusa]

It wasn't clear to me, but looking at some of the test cases it doesn't look like the [metadata] format records the hash type, e.g. sha256. Can we please record the hash algorithm with the hashes themselves? This allows for cryptographic agility in the future in order to upgrade to stronger hashes.

I'd also like to see sha512 used as the default algorithm since this is a new feature, and I don't think the extra length will cause any noticeable performance impacts.

[alexcrichton]

@luser

We'll have to nail down (and document) the process for making changes to vendored libraries

Yeah definitely, I'll try to write up documentation for all this, especially on the expected workflows. We can also chat more directly to make sure we're comfortable with it all.

@Ericson2314

If the section is called [source], perhaps any source including our existing git and path sources could be used as a mirror? Might that mean there would be no need for directory registries?

Unfortunately, no. To replace a source with another the checksum property (whether or not the source has checksums) must be the same. Registries have checksums, but git repositories and path sources do not. Vendored sources do, however.

@aneeshusa

Can we please record the hash algorithm with the hashes themselves? This allows for cryptographic agility in the future in order to upgrade to stronger hashes.

Yeah I was thinking we may want to do something like that. Of course we could also just say that if it starts with a bunch of unknown hex characters it's a sha256 hash, so it's not necessarily super pressing.

[bors]

☔ The latest upstream changes (presumably #2858) made this pull request unmergeable. Please resolve the merge conflicts.

[Ericson2314]

@alexcrichton Could those checksums be generated on the fly? If the checksum is generated by cargo publish (as opposed to the crates.io server) Cargo already contains the functionality!

[aneeshusa]

@alexcrichton Saying "if it starts with a bunch of unknown hex characters it's a sha256 hash" worries me.

• guessing formats has always been a source of vulnerabilities (see the libmagic CVEs for example), but this is especially bad for crypto; here is a blog post with another example.
• other hashes also use hex characters; how would you distinguish between them in the future? just using the length is not safe! SHA3 also has -256, -512, etc. variants, so these would be indistinguishable from the SHA2 versions without an explicit identifier.

Please be explicit here in the metadata format to allow for future changes without any guessing. The hash types used here will need to change in the future, so it would be better to avoid having to keep around a separate legacy "if there's no identifier yolo sha256" code path.

See pip for a good example of package verification.

[joshtriplett]

This looks excellent, and should work well for Debian packaging.

Regarding the local-registry index, could Cargo support reading the index from multiple files, in an index.d directory, if one exists? That would allow each crate to install its associated index entry. Without that, Debian would have to have a trigger that updates the index for every installed crate package.

Looking at the index format, it comes close to supporting this, but still requires a list of versions of each crate within a single file, which would make it harder to package each crate version separately. If each crate version could install a separate file, then packages of crate versions wouldn't need to run any code at installation time.

[Ericson2314]

@alexcrichton Moreover, for this NixOS case, the idea is what every source, not just crates.io, would need to be mirrored (nix would fetch all git repos, etc, so that Cargo would work without network access). So being able to mirror any source (not just registries), is crucial from our cases.

This is technically unrelated to being able to mirror a source with any sort of source, but as a practical matter, it would be much easier to replace a git source with its local clone than have to create a single-repo directory registry.

CC NixOS/nixpkgs#11144 @wycats ^ basically but we were talking about re nix.

[cuviper]

@joshtriplett Wouldn't the directory-registry work better for distro use? That's what I was leaning toward, then no index management is needed. The -devel packages would just drop their source into that shared root.

[bryteise]

@cuviper That makes more sense to me for a distro use case. The only thing this runs into trouble with is if people expect to carry different patches for the same build dependency across different packages which seems to be wrong in my mind.

[cuviper]

@bryteise I would expect those people to hash out their differences, just like they must for any other common dependency. IOW multiple sources shouldn't be dropping the same crate in that path, just one shared source that has to work for everyone.

[alexcrichton]

@Ericson2314

Could those checksums be generated on the fly?

No, I described this in "Directory registries" in the PR description.

So being able to mirror any source (not just registries), is crucial from our cases.

Yes, this is intended to be extensible for things like git repos as well, although it'd have to be specified once per git repo as well.

@joshtriplett

in an index.d directory, if one exists

Could you elaborate what you mean by this? I'm unfortunately not sure what index.d is :(. Right now it's kinda nice sharing all the index handling logic between the remote and local registries, and it'd be a bit of a shame to lose but perhaps not the worst!

[cuviper]

Usually "foo.d" is a directory of individual files to augment the main one. Like how there's /etc/profile and then everything in /etc/profile.d/ extends it. Packages can drop their own files in without having to merge in one common file.

[Ericson2314]

@alexcrichton you mean this paragraph?

Unlike local registries above we don't have a tarball to verify the crates.io checksum with, but each vendored dependency has metadata containing what it would have been. To further prevent modifications by accident, the metadata contains the checksum of each file which should prevent accidental local modifications and steer towards [replace] as the mechanism to edit dependencies if necessary.

I don't see why those files can't just be hashed on a fly. If the lockfile doesn't already exist, we are already assuming that the mirror reflects the original---precomputed hashes do not get around the fact that this is a matter of trust. If the lockfile does exist, the on-the-fly hashing is verified against that (Cargo computes the whole Merkel dag). I'm not saying we shouldn't have a directory repository---all that recursive hashing is slower---but I don't see a security problem.

[joshtriplett]

@cuviper @alexcrichton The compressed .crate files seemed nice to save space, but directory registries would work fine in distro packages too. The one downside would be the inability to match hashes directly with upstream, relying on the per-file hashes that (as I understand it) upstream doesn't directly provide. However, that property applies to every other package in the distribution, too, so it doesn't really matter.

The index.d concept was that each crate package could drop its portion of the index into a directory, rather than needing to have a single file that multiple crate packages would need to update. But if directory registries don't need that index at all, then that seems like the preferred alternative.

[joshtriplett]

@Ericson2314 It isn't a security problem; it's more that it stops casual "oh, I'll just patch the package" local hacks, which break the whole assumption of reproducible builds where version X of crate C always refers to the same crate contents.

[Ericson2314]

@joshtriplett Sure it's a not a real security problem. I'm just trying to get at what's worse about using the existing sources for mirrors. The only thing I could come up with is the slowness of hashing more things.

[cuviper]

Distros should be allowed to patch these things though!

[joshtriplett]

@cuviper Absolutely. But they shouldn't patch version X of crate C and still call it version X of crate C; at that point, it's, for instance, Debian package version X-2 (X-3, X-4, ...) of crate C.

[alexcrichton]

Yeah you're right in that we load up checksums for all the crates in a directory registry. Not doing so would require an index similar to the local registry index. The intent of directory sources are for vendoring, not distros, and in the case of vendoring you're going to load all the crates anyway to build them so the overhead shouldn't be much.

Also yeah, this isn't preventing any sort of "malicious activity" wrt directories. If you change a file you can change the checksum in the json metadata. The intent is to prevent accidental updates, not prevent malicious modifications. The core of the source replacement mechanism is that you replace a crate with the exact same code , just from a different location. If directory sources had no checksums at all then they couldn't provide that guarantee, but this allows them to at least provide a good enough guarantee along those lines for our purposes.

[alexcrichton]

Ok, @brson I believe I should have addressed all your comments and I've also pushed a commit containing documentation for this to go on doc.crates.io

[brson]

@bors r+

[bors]

📌 Commit 63ac9e1 has been approved by brson

[bors]

⌛ Testing commit 63ac9e1 with merge f09ef68...

[bors]

☀️ Test successful - cargo-cross-linux, cargo-linux-32, cargo-linux-64, cargo-mac-32, cargo-mac-64, cargo-win-gnu-32, cargo-win-gnu-64, cargo-win-msvc-32, cargo-win-msvc-64
Approved by: brson
Pushing f09ef68 to master...