Skip to content

Commit

Permalink
Merge pull request #2141 from integer32llc/alternative-registries
Browse files Browse the repository at this point in the history 
Add support to Cargo for alternative registries
  • Loading branch information
@aturon
aturon authored Sep 29, 2017
2 parents 09e34be + cbae4a3 commit 206318a
Showing 1 changed file with 365 additions and 0 deletions.
365 changes: 365 additions & 0 deletions text/2141-alternative-registries.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,365 @@
- Feature Name: cargo_alternative_registries
- Start Date: 2017-09-06
- RFC PR: https://github.com/rust-lang/rfcs/pull/2141
- Rust Issue: https://github.com/rust-lang/rust/issues/44931

# Summary
[summary]: #summary

This RFC proposes the addition of the support for alternative crates.io servers to be used
alongside the public crates.io server. This would allow users to publish crates to their own
private instance of crates.io, while still able to use the public instance of crates.io.

# Motivation
[motivation]: #motivation

Cargo currently has support for getting crates from a public server, which works well for open
source projects using Rust, however is problematic for closed source code. A workaround for this is
to use Git repositories to specify the packages, but that means that the helpful versioning and
discoverability that Cargo and crates.io provides is lost. We would like to change this such that
it is possible to have a local crates.io server which crates can be pushed to, while still making
use of the public crates.io server.

# Guide-level explanation
[guide-level-explanation]: #guide-level-explanation

## Registry definition specification
[registry-definition-specification]: #registry-definition-specification

We need a way to define what registries are valid for Cargo to pull from and publish to. For this
purpose, we propose that users would be able to define multiple registries in a [`.cargo/config`
file](http://doc.crates.io/config.html). This allows the user to specify the locations of
registries in one place, in a parent directory of all projects, rather than needing to configure
the registry location within each project's `Cargo.toml`. Once a registry has been configured with
a name, each `Cargo.toml` can use the registry name to refer to that registry.

Another benefit of using `.cargo/config` is that these files are not typically checked in to the
projects' source control. The registries might have credentials associated with them, which should
not be checked in. Separating the URLs and the use of the URLs in this way encourages good security
practices of not checking in credentials.

In order to tell Cargo about a registry other than crates.io, you can specify and name it in a
`.cargo/config` as follows, under the `registries` key:

```toml
[registries]
choose-a-name = "https://my-intranet:8080/index"
```

Instead of `choose-a-name`, place the name you'd like to use to refer to this registry in your
`Cargo.toml` files. The URL specified should contain the location of the registry index for this
registry; the registry format is specified in the [Registry Index Format Specification
section][registry-index-format-specification].

Alternatively, you can specify each registry as follows:

```toml
[registries.choose-a-name]
index = "https://my-intranet:8080/index"
```

If you need to specify authentication information such as a username or password to access a
registry's index, those should be specified in a `.cargo/credentials` file since it has more
restrictive file permissions than `.cargo/config`. Adding a username and password to
`.cargo/credentials` for a registry named `my-registry` would look like this:

```toml
[registries.my-registry]
username = "myusername"
password = "mypassword"
```

### CI

Because this system discourages checking in the registry configuration, the registry configuration
won't be immediately available to continuous integration systems like TravisCI. However, Cargo
currently supports configuring any key in `.cargo/config` using environment variables instead:

> Cargo can also be configured through environment variables in addition to the TOML syntax above.
> For each configuration key above of the form `foo.bar` the environment variable `CARGO_FOO_BAR`
> can also be used to define the value. For example the build.jobs key can also be defined by
> `CARGO_BUILD_JOBS`.
To configure TravisCI to use an alternate registry named `my-registry` for example, you can use
[Travis' encrypted environment variables feature](https://docs.travis-ci.com/user/environment-variables/#Defining-encrypted-variables-in-.travis.yml) to set:

`CARGO_REGISTRY_MY_REGISTRY_INDEX=https://my-intranet:8080/index`

## Using a dependency from another registry

*Note: this syntax will initially be implemented as an [unstable cargo
feature](https://github.com/rust-lang/cargo/pull/4433) available in nightly cargo only and
stabilized as it becomes ready.*

Once you've configured a registry (with a name, for example, `my-registry`) in `.cargo/config`, you
can specify that a dependency comes from an alternate registry by using the `registry` key:

```toml
[dependencies]
secret-crate = { version = "1.0", registry = "my-registry" }
```

## Publishing to another registry; preventing unwanted publishes

Today, Cargo allows you to add a key `publish = false` to your Cargo.toml to indicate that you do
not want to publish a crate anywhere. In order to specify that a crate should only be published to
a particular set of registries, this key will be extended to accept a list of registries that are
allowed with `cargo publish`:

```
publish = ["my-registry"]
```

If you run `cargo publish` without specifying an `--index` argument pointing to an allowed
registry, the command will fail. This prevents accidental publishes of private crates to crates.io,
for example.

Not having a `publish` key is equivalent to specifying `publish = true`, which means publishing to
crates.io is allowed. `publish = []` is equivalent to `publish = false`, meaning that publishing to
anywhere is disallowed.

## Running a minimal registry

The most minimal form of a registry that Cargo can use will consist of:

- A registry in the format specified in the [Registry index format specification
section][registry-index-format-specification], which contains a pointer to:
- A location containing the `.crate` files for the crates in the registry.

## Running a fully-featured registry

This RFC does not attempt to standardize or specify any of crates.io's APIs, but it should be
possible to take crates.io's codebase and run it along with a registry index in order to provide
crates.io's functionality as an alternate registry.

## Crates.io

Because crates.io's purpose is to be a reliable host for open source crates, crates that have
dependencies from registries other than crates.io will be rejected at publish time. Crates.io
cannot make availability guarantees about alternate registries, so much like git dependencies
today, publishing with dependencies from other registries won't be allowed.

In crates.io's codebase, we will add a configuration option that specifies a list of approved
alternate registry locations that dependencies may use. For private registries run using
crates.io's code, this will likely include the private registry itself plus crates.io, so that
private crates are allowed to depend on open source crates. Any crates with dependencies from
registries not specified in this configuration option will be rejected at publish time.

## Interaction with existing features

This RFC is not proposing any changes to the way [source
replacement](http://doc.crates.io/source-replacement.html) and
[cargo-vendor](https://crates.io/crates/cargo-vendor) work; everything proposed here should be
compatible with those.

Mirrors will still be required to serve exactly the same files (matched checksums) as the source
they're mirroring.

# Reference-level explanation
[reference-level-explanation]: #reference-level-explanation

## Registry index format specification
[registry-index-format-specification]: #registry-index-format-specification

Cargo needs to be able to get a registry index containing metadata for all crates and their
dependencies available from an alternate registry in order to perform offline version resolution.
The registry index for crates.io is available at
[https://github.com/rust-lang/crates.io-index](https://github.com/rust-lang/crates.io-index), and
this section aims to specify the format of this registry index so that other registries can provide
their own registry index that Cargo will understand.

This is version 1 of the registry index format specification. There may be other versions of the
specification someday. Along with a new specification version will be a plan for supporting
registries using the older specification and a migration plan for registries to upgrade the
specification version their index is using.

A valid registry index meets the following criteria:

- The registry index is stored in a git repository so that Cargo can efficiently fetch incremental
updates to the index.
- There will be a file at the top level named `config.json`. This file will be a valid JSON object
with the following keys:

```json
{
"dl": "https://my-crates-server.com/api/v1/crates",
"api": "https://my-crates-server.com/",
"allowed-registries": ["https://github.com/rust-lang/crates.io-index", "https://my-intranet:8080/index"]
}
```

The `dl` key is required specifies where Cargo can download the tarballs containing the source
files of the crates listed in the registry.

The `api` key is optional and specifies where Cargo can find the API server that provides the
same API functionality that crates.io does today, such as publishing and searching. Without the
`api` key, these features will not be available. This RFC is not attempting to standardize
crates.io's API in any way, although that could be a future enhancement.

The `allowed-registries` key is optional and specifies the other registries that crates in this
index are allowed to have dependencies on. The default will be nothing, which will mean only
crates that depend on other crates in the current registry are allowed. This is currently the
case for crates.io and will remain the case for crates.io going forward. Alternate registries
will probably want to add crates.io to this list.

- There will be a number of directories in the git repository.
- `1/` - holds files for all crates whose names have one letter.
- `2/` - holds files for all crates whose names have two letters.
- `3/` - holds files for all crates whose names have three letters.
- `aa/aa/` etc - for all crates whose names have four or more letters, their
files will be in a directory named with the first and second letters of
their name, then in a subdirectory named with the third and fourth letters
of their name. For example, a file for a crate named `sample` would be
found in `sa/mp/`.

- For each crate in the registry, there will be a file with the name of that crate in the directory
structure as specified above. The file will contain metadata about each version of the crate,
with one version per line. Each line will be valid JSON with, minimally, the keys as shown. More
keys may be added, but Cargo may ignore them. The contents of one line are pretty-printed here
for readability.

```json
{
"name": "my_serde",
"vers": "1.0.11",
"deps": [
{
"name": "serde",
"req": "^1.0",
"registry": "https://github.com/rust-lang/crates.io-index",
"features": [],
"optional": true,
"default_features": true,
"target": null,
"kind": "normal"
}
],
"cksum": "f7726f29ddf9731b17ff113c461e362c381d9d69433f79de4f3dd572488823e9",
"features": {
"default": [
"std"
],
"derive": [
"serde_derive"
],
"std": [

],
},
"yanked": false
}
```

The top-level keys for a crate are:

- `name`: the name of the crate
- `vers`: the version of the crate this row is describing
- `deps`: a list of all dependencies of this crate
- `cksum`: a SHA256 checksum of the tarball downloaded
- `features`: a list of the features available from this crate
- `yanked`: whether or not this version has been yanked

Within the `deps` list, each dependency should be listed as an item in the `deps` array with the
following keys:

- `name`: the name of the dependency
- `req`: the semver version requirement string on this dependency
- `registry`: **New to this RFC: the registry from which this crate is available**
- `features`: a list of the features available from this crate
- `optional`: whether this dependency is optional or not
- `default_features`: whether the parent uses the default features of this dependency or not
- `target`: on which target this dependency is needed
- `kind`: can be `normal`, `build`, or `dev` to be a regular dependency, a build-time
dependency, or a development dependency

If a dependency's registry is not specified, Cargo will assume the dependency can be located in the
current registry. By specifying the registry of a dependency in the index, cargo will have the
information it needs to fetch crate files from the registry indices involved without needing to
involve an API server.

## New command: `cargo generate-index-metadata`

Currently, the knowledge of how to create a file in the registry index format is spread between
Cargo and crates.io. This RFC proposes the addition of a Cargo command that would generate this
file locally for the current crate so that it can be added to the git repository using a mechanism
other than a server running crates.io's codebase.

## Related issues

In order to make working with multiple registries more convenient, we would also like to support:

- Adding a `cargo add-registry` command that could prompt for index URL and authentication
information and place the right information in the right format in the right files to make setup
for each user easier.
- [Being able to specify the API location rather than the index
location](https://github.com/rust-lang/cargo/issues/4208), so that, for example, you could
specify `https://host.company.com/api/cargo/private-repo` rather than
`https://github.com/host-company/cargo-index`. We do not want to *require* specifying the API
location, since some registries will choose not to have an API at all and only supply an index
and a location for crate files. This would require the API to have a way to tell Cargo where the
associated registry index is located.
- [Being able to save multiple tokens in
`.cargo/credentials`](https://github.com/rust-lang/cargo/issues/3365), one per registry, so that
people publishing to multiple registries don't need to log in over and over or specify tokens on
every publish.
- Being able to specify `--registry registry-name` for all Cargo commands that currently take
`--index`
- Being able to use a dependency under a different name. Alternate registries that are not mirrors
should be allowed to have crates with the same name as crates in any other registry, including
crates.io. In order to allow a crate to depend on both, say, the `http` crate from crates.io and
the `http` crate from a private registry, at least one will need to be renamed when listed as a
dependency in `Cargo.toml`. [RFC
2126](https://github.com/aturon/rfcs/blob/path-clarity/text/0000-path-clarity.md#basic-changes)
proposes this change as follows:

> Cargo will provide a new crate key for aliasing dependencies, so that e.g. users who want to
> use the `rand` crate but call it `random` instead can now write `random = { version = "0.3",
> crate = "rand" }`.
- Being able to use environment variables to specify values in `.cargo/credentials` in the same way
that you can use environment variables to specify values in `.cargo/config`
- For registries that don't require any authentication to access, such as public registries or
registries only accessible within a firewall, we could support a shorthand where the index
location (or API location when that is supported) is specified entirely within a crate dependency:

```toml
[dependencies]
my-crate = { version = "1.0", registry = "http://crate-mirror.org/index" }
```

In order to discourage/disallow credentials checked in to `Cargo.toml`, if the URL contains a
username or password, Cargo will deliberately remove it. If the registry is then inaccessible,
the error message will mention that usernames and passwords in URLs in `Cargo.toml` are not
allowed.

# Drawbacks
[drawbacks]: #drawbacks

Supporting alternative registries, and having multiple public registries, could fracture the
ecosystem. However, we feel that supporting private registries, and the Rust adoption that could
enable, outweighs the potential downsides of having multiple public registries.

# Rationale and Alternatives
[alternatives]: #alternatives

A [previous RFC](https://github.com/rust-lang/rfcs/pull/2006) proposed having the registry
information completely defined within `Cargo.toml` rather than using `.cargo/config`. This requires
repeating the same information multiple times for multiple projects, and encourages checking in
credentials that might be needed to access the registries. That RFC also didn't specify the format
for the registry index, which needs to be shared among all registries.

An alternative design could be to support specifying the registry URL in either `.cargo/config` or
`Cargo.toml`. This has the downsides of creating more choices for the user and potentially
encouraging poor practices such as checking credentials into a project's source control. The
implementation of this feature would also be more complex. The upside would be supporting
configuration in ways that would be more convenient in various situations.

# Unresolved questions
[unresolved]: #unresolved-questions

- Are the names of everything what we want?
- `cargo generate-index-metadata`?
- `registry = my-registry`?
- `publish-registries = []`?

- What kinds of authentication parameters do we need to support in `.cargo/credentials`?

0 comments on commit 206318a

Please sign in to comment.