Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RFC for support of alternate registries in cargo #2006

Closed
wants to merge 7 commits into from
120 changes: 120 additions & 0 deletions text/0000-alternate-registries.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,120 @@
- Feature Name: alternate-registries
- Start Date: 2017-05-23
- RFC PR: (leave this empty)
- Rust Issue: (leave this empty)

# Summary
[summary]: #summary

Adding support for alternative crates.io servers to be used alongside the public crates.io server.

# Motivation
[motivation]: #motivation

Currently cargo only has support for getting crates from a public server, this is fine for
open source projects using Rust, however for closed source code this is problematic. Currently
the only real option is to use Git repositories to specify the packages, but that means that
all of the nice versioning and discoverability that cargo and crates.io provides is lost. I
would like to change this so that it is possible to have a local crates.io server which private
crates can be pushed to, plus still be able to make use of the public crates.io server as well.

# Detailed design
[design]: #detailed-design

There are a number of different areas which will likely need to be tackled in order to fully
support a local crates.io server in an enterprise. Below are some key areas:

* Add support for alternate crates.io registries (this RFC)
* Provide support for caching crates used on a crates.io proxy
* Add support to crates.io to allow authentication with other OAuth providers than github
* Support for private storage of crates on crates.io server rather than publicly available on S3

Rather than trying to get everything agreed in a single RFC I would like for these to be
dealt with as separate proposals and use this RFC as a stepping stone to be able to
meaningfully start to use private registries.

See https://github.com/rust-lang/cargo/pull/4036 for the code changes which this proposal was based on.

## Cargo.toml config changes
The following changes for Cargo.toml are proposed as part of this RFC:
* Add an optional 'registry' field to be specified as part of the package dependencies
* Add an optional 'registry' field to be specified in the package definition
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if I want to be able to publish a crate to both a private registry and crates.io? For example, what if the workflow is that I publish prerelease versions to the private registry for internal testing, and then I publish polished versions to crates.io? I don't know if this is something that's wanted/needed, but it seems like it would be made inconvenient, if not impossible, by this implementation.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm... that's true. I can imagine such a feature being useful in CI environment.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be possible to do such a thing by not including the registry field and using a flag on your local crates.io server to allow a blank registry. I would not be intending to make such a change with this RFC though, I am trying to make this the minimum to unblock further work in this area.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO the simple solution here is to supply a --source (or --registry) flag which tells cargo which registry to publish to.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

supply a --source (or --registry) flag which tells cargo which registry to publish to

👍, in the given example the package could not specify a registry as the primary release channel is the default (or it could explicitly specify crates.io if it wanted), then setup the CI to do cargo publish --registry https://internal.registry/ to override it if the tag is a prerelease version.


Both of the registry entries take a value of a crates.io git repository. If one is not
provided then the default crates.io repository is assumed, this ensures that it is
back compatible with all current crates.

Below is an example of the change that we are proposing to make:

```
[package]
name = "registry-test"
version = "0.1.0"
authors = ["Christopher Swindle <christopher.swindle@metaswitch.com>"]
registry = "https://github.com/my_awesome_fork/crates.io-index"

[dependencies]
libc = { version = "*", registry = "https://github.com/my_awesome_fork/crates.io-index" }
serde_json = "1.0.1"
```

## Changes for alternative registries for dependencies
This boils down to a very simple change, where we previously setup the crate source for the
crates.io registry, we now just need to check if a registry is provided, if it has the crate
source is created using the registry URL, otherwise the crates.io server is used.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I'm installing all crates from a private registry, it seems annoying to have to specify that registry for every dependency. I think it would be better if there was a way to provide a default registry and then override that default per-crate if necessary.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My position is that there is a default registry and that is crates.io. I think keeping the registry with the dependency is a sensible approach as that makes it easier to copy dependencies from one Cargo.toml to another, but there is no stopping this from being done in a later stage.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am also in favor of "per dependency" config because of the "optimize for the reader" advice. When reading Cargo.toml, I would like to know from which registry each of the deficiencies comes, and per-dep config seems to be the easiest way to achieve that.

Adding a "default" registry would be backwards compatible, so we can skip it anyway at this stage :)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that the default is something you'll want to override on a per-dependency basis, but the current syntax seems pretty verbose and prone to typos. Something like this might be nicer?

[registries]
sone_name = "https://my_internal_registry.foobar"

[dependencies]
"some_name/foo" = "0.1"
libc = "0.3"


## Blocking requests to push to a registry
There are two parts to this, the first is a change to Cargo which checks if the registry
provided in the registry matches the host for the publish, if it does not it gets rejected.
The second part is a change to crates.io which will just reject the request to publish the
crate if the configured repository on the crates.io server does not match the registry
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This proposed change to crates.io means that the crates.io codebase needs to know the host where it is running, which it does not currently. That might be something you were implying here by "change to crates.io", but I just wanted to make sure you knew that currently the codebase doesn't have knowledge of where it's running.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It needs to know the git repository and compare that, which it already has, so I don't think that this is an issue.

specified in the package or dependencies.

## Allow publishing when referencing external dependencies
We still want to support private crates having dependencies on the public crates.io server,
so we propose relaxing a check which ensures that the source for a dependency matches the
registry. We propose that this only performs the check only if the dependency is not the
default registry, thus allowing private crates to reference public crates on crates.io.

## Making it easier for users using an alternate crates.io registry
When a user selects a specific crate the Cargo.toml fragment would be updated to include the
registry URL, thus allowing users to easily copy and paste into their projects Cargo.toml
file.

# How We Teach This
[how-we-teach-this]: #how-we-teach-this

The term alternative registry would seem the most appropriate to describe this feature.

In the first instance I think that the Cargo.toml format documentation is sufficient to
provide access to the feature. However, in time once more of the pieces fall into place
it would be useful having a guide on how to setup/administer a crates.io server in an
enterprise setting (similar to the initial mirroring documentation).

# Drawbacks
[drawbacks]: #drawbacks

Currently this design requires that when you want to push to the private crates.io server
you need to override the host and token, it would be possible to update cargo to support
multiple registries tokens which can be used to login.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rust-lang/cargo#3365 is relevant here :)

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree and that is something which I would like as well, but I wanted to try and get something in place, which can then be built on to add support for things like this.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI, we may get this rather soon because we are moving credentials anyway: rust-lang/cargo#3978

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice to know there is progress on that front as well :)


# Alternatives
[alternatives]: #alternatives

## Using a single server for cache and private registry
It was considered proposing a single crates.io server which performs both caching of crates.io,
plus has the ability to have crates pushed to it, however this has the following drawbacks:
* It requires crates.io to be able to combine two registries, or requires a radical change to the way crates.io works
* The current proposal could be extended to support this, if a caching server is added at a later stage

## Including registry definitions in a global location
We considered using a global configuration file (eg ~/.cargo/config) to allow a registry to

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think broad "we believe" statements like this are as truthful as they could be.

Future readers will no doubt read this as something everyone in this discussion agreed on which isn't the case.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, that was a reflection of discussions that I had within our company prior to submitting the RFC, not intended as a full reflection of the discussion here as I thought it would be useful to include.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cargo will look up the directory tree for .cargo/config files, so it is possible to have, for example, ~/some/path/my-project/.cargo/config rather than globally.

I think Cargo.toml is a better place for this information since it's intended to be checked in and shared while .cargo/config files are meant to be ignored and personal, but I wanted to make sure you knew about .cargo/config being an option.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know they can, but I just think that logically they belong in the Cargo.toml for the use cases I am trying to solve.

be specified, however this was ruled out on the basis that we believe that the registry to
use for dependencies is tightly linked to the project and hence it would be wrong to move
this into global configuration.

# Unresolved questions
[unresolved]: #unresolved-questions
As mentioned in the design section, this does not answer all of the questions for
supporting a private crates.io server, but it provides the first steps in that
direction, with the remaining areas considered out of scope for this RFC.