-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
RFC for support of alternate registries in cargo #2006
Changes from all commits
685157a
4d08fc2
9a35536
106440f
ddddb74
e2eac44
e61c4cb
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,174 @@ | ||
- Feature Name: alternate-registries | ||
- Start Date: 2017-05-23 | ||
- RFC PR: (leave this empty) | ||
- Rust Issue: (leave this empty) | ||
|
||
# Summary | ||
[summary]: #summary | ||
|
||
Adding support for alternative crates.io servers to be used alongside the public crates.io server. | ||
|
||
# Motivation | ||
[motivation]: #motivation | ||
|
||
Currently cargo only has support for getting crates from a public server, this is fine for | ||
open source projects using Rust, however for closed source code this is problematic. Currently | ||
the only real option is to use Git repositories to specify the packages, but that means that | ||
all of the nice versioning and discoverability that cargo and crates.io provides is lost. I | ||
would like to change this so that it is possible to have a local crates.io server which private | ||
crates can be pushed to, plus still be able to make use of the public crates.io server as well. | ||
|
||
# Detailed design | ||
[design]: #detailed-design | ||
|
||
There are a number of different areas which will likely need to be tackled in order to fully | ||
support a local crates.io server in an enterprise. Below are some key areas: | ||
|
||
* Add support for alternate crates.io registries (this RFC) | ||
* Provide support for caching crates used on a crates.io proxy | ||
* Add support to crates.io to allow authentication with other OAuth providers than github | ||
* Support for private storage of crates on crates.io server rather than publicly available on S3 | ||
|
||
Rather than trying to get everything agreed in a single RFC I would like for these to be | ||
dealt with as separate proposals and use this RFC as a stepping stone to be able to | ||
meaningfully start to use private registries. | ||
|
||
See https://github.com/rust-lang/cargo/pull/4036 for the code changes which this proposal was based on. | ||
|
||
## Cargo.toml config changes | ||
The following changes for Cargo.toml are proposed as part of this RFC: | ||
* Add an optional 'registry' field to be specified as part of the package dependencies | ||
* Add an optional 'registry' field to be specified in the package definition | ||
|
||
Both of the registry entries take a value of a crates.io web server. If one is not | ||
provided then the default crates.io URL is assumed, this ensures that it is | ||
back compatible with all current crates. | ||
|
||
Below is an example of the change that we are proposing to make: | ||
|
||
``` | ||
[package] | ||
name = "registry-test" | ||
version = "0.1.0" | ||
authors = ["Christopher Swindle <christopher.swindle@metaswitch.com>"] | ||
registry = "https://www.my_awesome_fork.com/" | ||
|
||
[dependencies] | ||
libc = { version = "*", registry = "https://www.my_awesome_fork.com/" } | ||
serde_json = "1.0.1" | ||
``` | ||
|
||
## Mapping Between Registry and Git Repository | ||
Currently the Git repository is used internally within Cargo, in order for Cargo to map | ||
between the servers URL and a Git respository for the crates.io web server, the server | ||
exposes a set endpoint which provides the Git repository to use, for example: | ||
|
||
``` | ||
http://crates.io/git | ||
``` | ||
|
||
This will just return the Git repository to use, which the server already knows. When cargo | ||
needs to perform an action on an alternate registry it just performs a GET request | ||
on the URL and then uses the Git repository returned. | ||
|
||
## Dependency resolution | ||
This RFC proposes going to an alternative registry for a dependency only if the registry key | ||
is present in the dependency. This means that in situations where there is the same name | ||
crate on both crates.io and the alternate registry, if no registry is provided it will | ||
use the one from crates.io. There are valid situations where people may wish to override | ||
a particular crate with an alternate one, however the existing source replacement feature | ||
seems a better fit to solve that scenario. | ||
|
||
## Crate naming on alternate servers | ||
It would be sensible, on an alternate server, for users not to publish using the same | ||
name as a public crate as that will cause issues if someone need to use a public and private | ||
crate with the same name. In order to minimise the risk of this happening I propose | ||
that an optional field is added to the crates.io config which allows a prefix to be configured, | ||
when a crate is published, the prefix is checked and the request rejected with a sensible error | ||
if the crate name does not match. | ||
|
||
## Changes for alternative registries for dependencies | ||
This boils down to a very simple change, where we previously setup the crate source for the | ||
crates.io registry, we now just need to check if a registry is provided, if it has the crate | ||
source is created using the registry URL, otherwise the crates.io server is used. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. If I'm installing all crates from a private registry, it seems annoying to have to specify that registry for every dependency. I think it would be better if there was a way to provide a default registry and then override that default per-crate if necessary. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. My position is that there is a default registry and that is crates.io. I think keeping the registry with the dependency is a sensible approach as that makes it easier to copy dependencies from one Cargo.toml to another, but there is no stopping this from being done in a later stage. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I am also in favor of "per dependency" config because of the "optimize for the reader" advice. When reading Cargo.toml, I would like to know from which registry each of the deficiencies comes, and per-dep config seems to be the easiest way to achieve that. Adding a "default" registry would be backwards compatible, so we can skip it anyway at this stage :) There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I agree that the default is something you'll want to override on a per-dependency basis, but the current syntax seems pretty verbose and prone to typos. Something like this might be nicer? [registries]
sone_name = "https://my_internal_registry.foobar"
[dependencies]
"some_name/foo" = "0.1"
libc = "0.3" |
||
|
||
### Index files changes | ||
As Cargo requires the index file to include all the dependencies, the crates.io index file | ||
format is updated to include the registry in the dependency. The registry is an optional field, | ||
where by default it is None, and will only be set when using an alternate crate server. The | ||
official public crates.io server will block any publish requests which contain a registry | ||
in dependencies, so for crates.io this will always be set to None. | ||
|
||
Validation of the depdencies on crates.io is also updated so that local registry crates are | ||
checked for their existance on the local registry, they will not however validate any | ||
external dependencies, instead they will be assumed to be valid. | ||
|
||
## Blocking requests to push to a registry | ||
Cargo will by try and publish to the registry that is provided in the Cargo.toml (if one is | ||
provided), if the registry has been override and it does not match the entry in Cargo.toml it will | ||
be rejected. The second part is a change to crates.io which will just reject the request to | ||
publish the crate if the configured registry on the crates.io server does not match the | ||
registry specified in the package or dependencies. | ||
|
||
## Allow publishing when referencing external dependencies | ||
We still want to support private crates having dependencies on the public crates.io server, | ||
so we propose relaxing a check which ensures that the source for a dependency matches the | ||
registry. We propose that this only performs the check only if the dependency is not the | ||
default registry, thus allowing private crates to reference public crates on crates.io. | ||
|
||
## Making it easier for users using an alternate crates.io registry | ||
When a user selects a specific crate the Cargo.toml fragment would be updated to include the | ||
registry URL, thus allowing users to easily copy and paste into their projects Cargo.toml | ||
file. Below is an example of how this might look: | ||
|
||
<img src="http://i.imgur.com/znMMwAc.jpg" alt="Example layout of crates.io with URL in fragment" /> | ||
|
||
# How We Teach This | ||
[how-we-teach-this]: #how-we-teach-this | ||
|
||
The term alternative registry would seem the most appropriate to describe this feature. | ||
|
||
In the first instance I think that the Cargo.toml format documentation is sufficient to | ||
provide access to the feature. However, in time once more of the pieces fall into place | ||
it would be useful having a guide on how to setup/administer a crates.io server in an | ||
enterprise setting (similar to the initial mirroring documentation). | ||
|
||
# Drawbacks | ||
[drawbacks]: #drawbacks | ||
|
||
Currently this design requires that when you want to push to the private crates.io server | ||
you need to override the host and token, it would be possible to update cargo to support | ||
multiple registries tokens which can be used to login. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. rust-lang/cargo#3365 is relevant here :) There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I agree and that is something which I would like as well, but I wanted to try and get something in place, which can then be built on to add support for things like this. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. FYI, we may get this rather soon because we are moving credentials anyway: rust-lang/cargo#3978 There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Nice to know there is progress on that front as well :) |
||
|
||
# Alternatives | ||
[alternatives]: #alternatives | ||
|
||
## Using a single server for cache and private registry | ||
It was considered proposing a single crates.io server which performs both caching of crates.io, | ||
plus has the ability to have crates pushed to it, however this has the following drawbacks: | ||
* It requires crates.io to be able to combine two registries, or requires a radical change to the way crates.io works | ||
* The current proposal could be extended to support this, if a caching server is added at a later stage | ||
|
||
## Including registry definitions in a global location | ||
We considered using a global configuration file (eg ~/.cargo/config) to allow a registry to | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Cargo will look up the directory tree for I think Cargo.toml is a better place for this information since it's intended to be checked in and shared while There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I know they can, but I just think that logically they belong in the Cargo.toml for the use cases I am trying to solve. |
||
be specified, however this was ruled out on the basis that we believe that the registry to | ||
use for dependencies is tightly linked to the project and hence it would be wrong to move | ||
this into global configuration. | ||
|
||
## Validating external dependencies | ||
It would be possible to check that external dependencies exist, however I think that this | ||
would need to be optional as there could be situations where the alternate crates.io | ||
server would be unable to contact the external dependencies and hence would not be | ||
possible. This is certainly something that could be added at a later stage if there is | ||
sufficient demand. | ||
|
||
# Unresolved questions | ||
[unresolved]: #unresolved-questions | ||
As mentioned in the design section, this RFC does not answer all of the questions for | ||
supporting a private crates.io server, but it provides the first steps in that direction, | ||
with the remaining areas considered out of scope for this RFC. However, the following issues | ||
will need to be solved before this RFC can be fully implemented, although not | ||
necessarily directly as part of this RFC: | ||
|
||
* There needs to be a mechanism for unstable features in cargo | ||
* A public API needs to be available for the registry format prior to this feature being available in stable builds |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What if I want to be able to publish a crate to both a private registry and crates.io? For example, what if the workflow is that I publish prerelease versions to the private registry for internal testing, and then I publish polished versions to crates.io? I don't know if this is something that's wanted/needed, but it seems like it would be made inconvenient, if not impossible, by this implementation.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmm... that's true. I can imagine such a feature being useful in CI environment.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It would be possible to do such a thing by not including the registry field and using a flag on your local crates.io server to allow a blank registry. I would not be intending to make such a change with this RFC though, I am trying to make this the minimum to unblock further work in this area.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
IMO the simple solution here is to supply a --source (or --registry) flag which tells cargo which registry to publish to.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍, in the given example the package could not specify a registry as the primary release channel is the default (or it could explicitly specify
crates.io
if it wanted), then setup the CI to docargo publish --registry https://internal.registry/
to override it if the tag is a prerelease version.