- Feature Name: cargo_alternative_registries
- Start Date: 2017-09-06
- RFC PR: rust-lang/rfcs#2141
- Rust Issue: rust-lang/rust#44931
Summary
This RFC proposes the addition of the support for alternative crates.io servers to be used alongside the public crates.io server. This would allow users to publish crates to their own private instance of crates.io, while still able to use the public instance of crates.io.
Motivation
Cargo currently has support for getting crates from a public server, which works well for open source projects using Rust, however is problematic for closed source code. A workaround for this is to use Git repositories to specify the packages, but that means that the helpful versioning and discoverability that Cargo and crates.io provides is lost. We would like to change this such that it is possible to have a local crates.io server which crates can be pushed to, while still making use of the public crates.io server.
Guide-level explanation
Registry definition specification
We need a way to define what registries are valid for Cargo to pull from and publish to. For this
purpose, we propose that users would be able to define multiple registries in a .cargo/config
file. This allows the user to specify the locations of
registries in one place, in a parent directory of all projects, rather than needing to configure
the registry location within each project’s Cargo.toml
. Once a registry has been configured with
a name, each Cargo.toml
can use the registry name to refer to that registry.
Another benefit of using .cargo/config
is that these files are not typically checked in to the
projects’ source control. The registries might have credentials associated with them, which should
not be checked in. Separating the URLs and the use of the URLs in this way encourages good security
practices of not checking in credentials.
In order to tell Cargo about a registry other than crates.io, you can specify and name it in a
.cargo/config
as follows, under the registries
key:
[registries]
choose-a-name = "https://my-intranet:8080/index"
Instead of choose-a-name
, place the name you’d like to use to refer to this registry in your
Cargo.toml
files. The URL specified should contain the location of the registry index for this
registry; the registry format is specified in the Registry Index Format Specification
section.
Alternatively, you can specify each registry as follows:
[registries.choose-a-name]
index = "https://my-intranet:8080/index"
If you need to specify authentication information such as a username or password to access a
registry’s index, those should be specified in a .cargo/credentials
file since it has more
restrictive file permissions than .cargo/config
. Adding a username and password to
.cargo/credentials
for a registry named my-registry
would look like this:
[registries.my-registry]
username = "myusername"
password = "mypassword"
CI
Because this system discourages checking in the registry configuration, the registry configuration
won’t be immediately available to continuous integration systems like TravisCI. However, Cargo
currently supports configuring any key in .cargo/config
using environment variables instead:
Cargo can also be configured through environment variables in addition to the TOML syntax above. For each configuration key above of the form
foo.bar
the environment variableCARGO_FOO_BAR
can also be used to define the value. For example the build.jobs key can also be defined byCARGO_BUILD_JOBS
.
To configure TravisCI to use an alternate registry named my-registry
for example, you can use
Travis’ encrypted environment variables feature to set:
CARGO_REGISTRIES_MY_REGISTRY_INDEX=https://my-intranet:8080/index
Using a dependency from another registry
Note: this syntax will initially be implemented as an unstable cargo feature available in nightly cargo only and stabilized as it becomes ready.
Once you’ve configured a registry (with a name, for example, my-registry
) in .cargo/config
, you
can specify that a dependency comes from an alternate registry by using the registry
key:
[dependencies]
secret-crate = { version = "1.0", registry = "my-registry" }
Publishing to another registry; preventing unwanted publishes
Today, Cargo allows you to add a key publish = false
to your Cargo.toml to indicate that you do
not want to publish a crate anywhere. In order to specify that a crate should only be published to
a particular set of registries, this key will be extended to accept a list of registries that are
allowed with cargo publish
:
publish = ["my-registry"]
If you run cargo publish
without specifying an --index
argument pointing to an allowed
registry, the command will fail. This prevents accidental publishes of private crates to crates.io,
for example.
Not having a publish
key is equivalent to specifying publish = true
, which means publishing to
crates.io is allowed. publish = []
is equivalent to publish = false
, meaning that publishing to
anywhere is disallowed.
Running a minimal registry
The most minimal form of a registry that Cargo can use will consist of:
- A registry in the format specified in the Registry index format specification section, which contains a pointer to:
- A location containing the
.crate
files for the crates in the registry.
Running a fully-featured registry
This RFC does not attempt to standardize or specify any of crates.io’s APIs, but it should be possible to take crates.io’s codebase and run it along with a registry index in order to provide crates.io’s functionality as an alternate registry.
Crates.io
Because crates.io’s purpose is to be a reliable host for open source crates, crates that have dependencies from registries other than crates.io will be rejected at publish time. Crates.io cannot make availability guarantees about alternate registries, so much like git dependencies today, publishing with dependencies from other registries won’t be allowed.
In crates.io’s codebase, we will add a configuration option that specifies a list of approved alternate registry locations that dependencies may use. For private registries run using crates.io’s code, this will likely include the private registry itself plus crates.io, so that private crates are allowed to depend on open source crates. Any crates with dependencies from registries not specified in this configuration option will be rejected at publish time.
Interaction with existing features
This RFC is not proposing any changes to the way source replacement and cargo-vendor work; everything proposed here should be compatible with those.
Mirrors will still be required to serve exactly the same files (matched checksums) as the source they’re mirroring.
Reference-level explanation
Registry index format specification
Cargo needs to be able to get a registry index containing metadata for all crates and their dependencies available from an alternate registry in order to perform offline version resolution. The registry index for crates.io is available at https://github.com/rust-lang/crates.io-index, and this section aims to specify the format of this registry index so that other registries can provide their own registry index that Cargo will understand.
This is version 1 of the registry index format specification. There may be other versions of the specification someday. Along with a new specification version will be a plan for supporting registries using the older specification and a migration plan for registries to upgrade the specification version their index is using.
A valid registry index meets the following criteria:
-
The registry index is stored in a git repository so that Cargo can efficiently fetch incremental updates to the index.
-
There will be a file at the top level named
config.json
. This file will be a valid JSON object with the following keys:{ "dl": "https://my-crates-server.com/api/v1/crates/{crate}/{version}/download", "api": "https://my-crates-server.com/", "allowed-registries": ["https://github.com/rust-lang/crates.io-index", "https://my-intranet:8080/index"] }
The
dl
key is required and specifies where Cargo can download the tarballs containing the source files of the crates listed in the registry. It is templated by the strings{crate}
and{version}
which are replaced with the name and version of the crate to download, respectively.The
api
key is optional and specifies where Cargo can find the API server that provides the same API functionality that crates.io does today, such as publishing and searching. Without theapi
key, these features will not be available. This RFC is not attempting to standardize crates.io’s API in any way, although that could be a future enhancement.The
allowed-registries
key is optional and specifies the other registries that crates in this index are allowed to have dependencies on. The default will be nothing, which will mean only crates that depend on other crates in the current registry are allowed. This is currently the case for crates.io and will remain the case for crates.io going forward. Alternate registries will probably want to add crates.io to this list. -
There will be a number of directories in the git repository.
1/
- holds files for all crates whose names have one letter.2/
- holds files for all crates whose names have two letters.3/a
etc - for all crates whose names have three letters, their files will be in a directory named3
, then a subdirectory named with the first letter of their name.aa/aa/
etc - for all crates whose names have four or more letters, their files will be in a directory named with the first and second letters of their name, then in a subdirectory named with the third and fourth letters of their name. For example, a file for a crate namedsample
would be found insa/mp/
.
-
For each crate in the registry, there will be a file with the name of that crate in the directory structure as specified above. The file will contain metadata about each version of the crate, with one version per line. Each line will be valid JSON with, minimally, the keys as shown. More keys may be added, but Cargo may ignore them. The contents of one line are pretty-printed here for readability.
{ "name": "my_serde", "vers": "1.0.11", "deps": [ { "name": "serde", "req": "^1.0", "registry": "https://github.com/rust-lang/crates.io-index", "features": [], "optional": true, "default_features": true, "target": null, "kind": "normal" } ], "cksum": "f7726f29ddf9731b17ff113c461e362c381d9d69433f79de4f3dd572488823e9", "features": { "default": [ "std" ], "derive": [ "serde_derive" ], "std": [ ], }, "yanked": false }
The top-level keys for a crate are:
name
: the name of the cratevers
: the version of the crate this row is describingdeps
: a list of all dependencies of this cratecksum
: a SHA256 checksum of the tarball downloadedfeatures
: a list of the features available from this crateyanked
: whether or not this version has been yanked
Within the
deps
list, each dependency should be listed as an item in thedeps
array with the following keys:name
: the name of the dependencyreq
: the semver version requirement string on this dependencyregistry
: New to this RFC: the registry from which this crate is availablefeatures
: a list of the features available from this crateoptional
: whether this dependency is optional or notdefault_features
: whether the parent uses the default features of this dependency or nottarget
: on which target this dependency is neededkind
: can benormal
,build
, ordev
to be a regular dependency, a build-time dependency, or a development dependency. Note: this is a required field, but a small number of entries exist in the crates.io index with either a missing or nullkind
field due to implementation bugs.
If a dependency’s registry is not specified, Cargo will assume the dependency can be located in the current registry. By specifying the registry of a dependency in the index, cargo will have the information it needs to fetch crate files from the registry indices involved without needing to involve an API server.
New command: cargo generate-index-metadata
Currently, the knowledge of how to create a file in the registry index format is spread between Cargo and crates.io. This RFC proposes the addition of a Cargo command that would generate this file locally for the current crate so that it can be added to the git repository using a mechanism other than a server running crates.io’s codebase.
Related issues
In order to make working with multiple registries more convenient, we would also like to support:
-
Adding a
cargo add-registry
command that could prompt for index URL and authentication information and place the right information in the right format in the right files to make setup for each user easier. -
Being able to specify the API location rather than the index location, so that, for example, you could specify
https://host.company.com/api/cargo/private-repo
rather thanhttps://github.com/host-company/cargo-index
. We do not want to require specifying the API location, since some registries will choose not to have an API at all and only supply an index and a location for crate files. This would require the API to have a way to tell Cargo where the associated registry index is located. -
Being able to save multiple tokens in
.cargo/credentials
, one per registry, so that people publishing to multiple registries don’t need to log in over and over or specify tokens on every publish. -
Being able to specify
--registry registry-name
for all Cargo commands that currently take--index
-
Being able to use a dependency under a different name. Alternate registries that are not mirrors should be allowed to have crates with the same name as crates in any other registry, including crates.io. In order to allow a crate to depend on both, say, the
http
crate from crates.io and thehttp
crate from a private registry, at least one will need to be renamed when listed as a dependency inCargo.toml
. RFC 2126 proposes this change as follows:Cargo will provide a new crate key for aliasing dependencies, so that e.g. users who want to use the
rand
crate but call itrandom
instead can now writerandom = { version = "0.3", crate = "rand" }
. -
Being able to use environment variables to specify values in
.cargo/credentials
in the same way that you can use environment variables to specify values in.cargo/config
-
For registries that don’t require any authentication to access, such as public registries or registries only accessible within a firewall, we could support a shorthand where the index location (or API location when that is supported) is specified entirely within a crate dependency:
[dependencies] my-crate = { version = "1.0", registry = "http://crate-mirror.org/index" }
In order to discourage/disallow credentials checked in to
Cargo.toml
, if the URL contains a username or password, Cargo will deliberately remove it. If the registry is then inaccessible, the error message will mention that usernames and passwords in URLs inCargo.toml
are not allowed.
Drawbacks
Supporting alternative registries, and having multiple public registries, could fracture the ecosystem. However, we feel that supporting private registries, and the Rust adoption that could enable, outweighs the potential downsides of having multiple public registries.
Rationale and Alternatives
A previous RFC proposed having the registry
information completely defined within Cargo.toml
rather than using .cargo/config
. This requires
repeating the same information multiple times for multiple projects, and encourages checking in
credentials that might be needed to access the registries. That RFC also didn’t specify the format
for the registry index, which needs to be shared among all registries.
An alternative design could be to support specifying the registry URL in either .cargo/config
or
Cargo.toml
. This has the downsides of creating more choices for the user and potentially
encouraging poor practices such as checking credentials into a project’s source control. The
implementation of this feature would also be more complex. The upside would be supporting
configuration in ways that would be more convenient in various situations.
Unresolved questions
-
Are the names of everything what we want?
cargo generate-index-metadata
?registry = my-registry
?publish-registries = []
?
-
What kinds of authentication parameters do we need to support in
.cargo/credentials
?