RFC for support of alternate registries in cargo #2006
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,120 @@ | ||
| - Feature Name: alternate-registries | ||
| - Start Date: 2017-05-23 | ||
| - RFC PR: (leave this empty) | ||
| - Rust Issue: (leave this empty) | ||
|
|
||
| # Summary | ||
| [summary]: #summary | ||
|
|
||
| Adding support for alternative crates.io servers to be used alongside the public crates.io server. | ||
|
|
||
| # Motivation | ||
| [motivation]: #motivation | ||
|
|
||
| Currently cargo only has support for getting crates from a public server, this is fine for | ||
| open source projects using Rust, however for closed source code this is problematic. Currently | ||
| the only real option is to use Git repositories to specify the packages, but that means that | ||
| all of the nice versioning and discoverability that cargo and crates.io provides is lost. I | ||
| would like to change this so that it is possible to have a local crates.io server which private | ||
| crates can be pushed to, plus still be able to make use of the public crates.io server as well. | ||
|
|
||
| # Detailed design | ||
| [design]: #detailed-design | ||
|
|
||
| There are a number of different areas which will likely need to be tackled in order to fully | ||
| support a local crates.io server in an enterprise. Below are some key areas: | ||
|
|
||
| * Add support for alternate crates.io registries (this RFC) | ||
| * Provide support for caching crates used on a crates.io proxy | ||
| * Add support to crates.io to allow authentication with other OAuth providers than github | ||
| * Support for private storage of crates on crates.io server rather than publicly available on S3 | ||
|
|
||
| Rather than trying to get everything agreed in a single RFC I would like for these to be | ||
| dealt with as separate proposals and use this RFC as a stepping stone to be able to | ||
| meaningfully start to use private registries. | ||
|
|
||
| See https://github.com/rust-lang/cargo/pull/4036 for the code changes which this proposal was based on. | ||
|
|
||
| ## Cargo.toml config changes | ||
| The following changes for Cargo.toml are proposed as part of this RFC: | ||
| * Add an optional 'registry' field to be specified as part of the package dependencies | ||
| * Add an optional 'registry' field to be specified in the package definition | ||
|
|
||
| Both of the registry entries take a value of a crates.io git repository. If one is not | ||
| provided then the default crates.io repository is assumed, this ensures that it is | ||
| back compatible with all current crates. | ||
|
|
||
| Below is an example of the change that we are proposing to make: | ||
|
|
||
| ``` | ||
| [package] | ||
| name = "registry-test" | ||
| version = "0.1.0" | ||
| authors = ["Christopher Swindle <christopher.swindle@metaswitch.com>"] | ||
| registry = "https://github.com/my_awesome_fork/crates.io-index" | ||
|
|
||
| [dependencies] | ||
| libc = { version = "*", registry = "https://github.com/my_awesome_fork/crates.io-index" } | ||
| serde_json = "1.0.1" | ||
| ``` | ||
|
|
||
| ## Changes for alternative registries for dependencies | ||
| This boils down to a very simple change, where we previously setup the crate source for the | ||
| crates.io registry, we now just need to check if a registry is provided, if it has the crate | ||
| source is created using the registry URL, otherwise the crates.io server is used. | ||
|
There was a problem hiding this comment. If I'm installing all crates from a private registry, it seems annoying to have to specify that registry for every dependency. I think it would be better if there was a way to provide a default registry and then override that default per-crate if necessary. There was a problem hiding this comment. My position is that there is a default registry and that is crates.io. I think keeping the registry with the dependency is a sensible approach as that makes it easier to copy dependencies from one Cargo.toml to another, but there is no stopping this from being done in a later stage. There was a problem hiding this comment. I am also in favor of "per dependency" config because of the "optimize for the reader" advice. When reading Cargo.toml, I would like to know from which registry each of the deficiencies comes, and per-dep config seems to be the easiest way to achieve that. Adding a "default" registry would be backwards compatible, so we can skip it anyway at this stage :) There was a problem hiding this comment. I agree that the default is something you'll want to override on a per-dependency basis, but the current syntax seems pretty verbose and prone to typos. Something like this might be nicer? [registries]
sone_name = "https://my_internal_registry.foobar"
[dependencies]
"some_name/foo" = "0.1"
libc = "0.3" |
||
|
|
||
| ## Blocking requests to push to a registry | ||
| There are two parts to this, the first is a change to Cargo which checks if the registry | ||
| provided in the registry matches the host for the publish, if it does not it gets rejected. | ||
| The second part is a change to crates.io which will just reject the request to publish the | ||
| crate if the configured repository on the crates.io server does not match the registry | ||
|
There was a problem hiding this comment. This proposed change to crates.io means that the crates.io codebase needs to know the host where it is running, which it does not currently. That might be something you were implying here by "change to crates.io", but I just wanted to make sure you knew that currently the codebase doesn't have knowledge of where it's running. There was a problem hiding this comment. It needs to know the git repository and compare that, which it already has, so I don't think that this is an issue. |
||
| specified in the package or dependencies. | ||
|
|
||
| ## Allow publishing when referencing external dependencies | ||
| We still want to support private crates having dependencies on the public crates.io server, | ||
| so we propose relaxing a check which ensures that the source for a dependency matches the | ||
| registry. We propose that this only performs the check only if the dependency is not the | ||
| default registry, thus allowing private crates to reference public crates on crates.io. | ||
|
|
||
| ## Making it easier for users using an alternate crates.io registry | ||
| When a user selects a specific crate the Cargo.toml fragment would be updated to include the | ||
| registry URL, thus allowing users to easily copy and paste into their projects Cargo.toml | ||
| file. | ||
|
|
||
| # How We Teach This | ||
| [how-we-teach-this]: #how-we-teach-this | ||
|
|
||
| The term alternative registry would seem the most appropriate to describe this feature. | ||
|
|
||
| In the first instance I think that the Cargo.toml format documentation is sufficient to | ||
| provide access to the feature. However, in time once more of the pieces fall into place | ||
| it would be useful having a guide on how to setup/administer a crates.io server in an | ||
| enterprise setting (similar to the initial mirroring documentation). | ||
|
|
||
| # Drawbacks | ||
| [drawbacks]: #drawbacks | ||
|
|
||
| Currently this design requires that when you want to push to the private crates.io server | ||
| you need to override the host and token, it would be possible to update cargo to support | ||
| multiple registries tokens which can be used to login. | ||
|
There was a problem hiding this comment. rust-lang/cargo#3365 is relevant here :) There was a problem hiding this comment. I agree and that is something which I would like as well, but I wanted to try and get something in place, which can then be built on to add support for things like this. There was a problem hiding this comment. FYI, we may get this rather soon because we are moving credentials anyway: rust-lang/cargo#3978 There was a problem hiding this comment. Nice to know there is progress on that front as well :) |
||
|
|
||
| # Alternatives | ||
| [alternatives]: #alternatives | ||
|
|
||
| ## Using a single server for cache and private registry | ||
| It was considered proposing a single crates.io server which performs both caching of crates.io, | ||
| plus has the ability to have crates pushed to it, however this has the following drawbacks: | ||
| * It requires crates.io to be able to combine two registries, or requires a radical change to the way crates.io works | ||
| * The current proposal could be extended to support this, if a caching server is added at a later stage | ||
|
|
||
| ## Including registry definitions in a global location | ||
| We considered using a global configuration file (eg ~/.cargo/config) to allow a registry to | ||
|
There was a problem hiding this comment. I don't think broad "we believe" statements like this are as truthful as they could be. Future readers will no doubt read this as something everyone in this discussion agreed on which isn't the case. There was a problem hiding this comment. Sorry, that was a reflection of discussions that I had within our company prior to submitting the RFC, not intended as a full reflection of the discussion here as I thought it would be useful to include. There was a problem hiding this comment. Cargo will look up the directory tree for I think Cargo.toml is a better place for this information since it's intended to be checked in and shared while There was a problem hiding this comment. I know they can, but I just think that logically they belong in the Cargo.toml for the use cases I am trying to solve. |
||
| be specified, however this was ruled out on the basis that we believe that the registry to | ||
| use for dependencies is tightly linked to the project and hence it would be wrong to move | ||
| this into global configuration. | ||
|
|
||
| # Unresolved questions | ||
| [unresolved]: #unresolved-questions | ||
| As mentioned in the design section, this does not answer all of the questions for | ||
| supporting a private crates.io server, but it provides the first steps in that | ||
| direction, with the remaining areas considered out of scope for this RFC. | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What if I want to be able to publish a crate to both a private registry and crates.io? For example, what if the workflow is that I publish prerelease versions to the private registry for internal testing, and then I publish polished versions to crates.io? I don't know if this is something that's wanted/needed, but it seems like it would be made inconvenient, if not impossible, by this implementation.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmm... that's true. I can imagine such a feature being useful in CI environment.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It would be possible to do such a thing by not including the registry field and using a flag on your local crates.io server to allow a blank registry. I would not be intending to make such a change with this RFC though, I am trying to make this the minimum to unblock further work in this area.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
IMO the simple solution here is to supply a --source (or --registry) flag which tells cargo which registry to publish to.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍, in the given example the package could not specify a registry as the primary release channel is the default (or it could explicitly specify
crates.ioif it wanted), then setup the CI to docargo publish --registry https://internal.registry/to override it if the tag is a prerelease version.