Add support for a nightly-alt toolchain

[SimonSapin]

Edit: rust-lang-ci is ephemeral, so the proposal is now to add one nightly-alt channel adressed by date like nightly.

---

The builds made for testing every Rust pull request before merging are made available at:

• https://s3.amazonaws.com/rust-lang-ci/rustc-builds/$SHA1/rust-nightly-$HOST_TRIPLE.tar.gz
• https://s3.amazonaws.com/rust-lang-ci/rustc-builds-alt/$SHA1/rust-nightly-$HOST_TRIPLE.tar.gz

… where $SHA1 is the full hex hash of a merge commit made by @bors.

The latter was added in rust-lang/rust#39754. It includes a smaller number of platforms, and disables LLVM assertions which weakens compiler testing but improves compiler performance.

Servo has been using these builds. They prove useful both for the performance improvement and for testing for example a bug fix that has been merged without waiting for the next Nightly (which sometimes takes several additional days).

Servo does this with custom rustup-like Python scripts. These builds can be used in rustup with rustup toolchain link FOO /path/to/extracted/tarball, but downloading them and extracting them manually can get tedious. Automating this approaches re-inventing rustup, so it would be nicer if rustup could support it directly.

master and master-alt could be added to the toolchains rustup knows about out of the box like stable and nightly. The hash of the current latest commit can be found in https://api.github.com/repos/rust-lang/rust/git/refs/heads/master. These toolchains would also support specifying a precise hash, like stable and nightly support a version number and a date:

rustup update stable-1.17.0
rustup update nightly-2017-05-06
rustup update master-alt-70198a0a44633c7c9d14fce2159c1f750491287b

@brson How does this sound?

[Diggsey]

@alexcrichton said this would be difficult, but it would kill two birds with one stone if we could produce manifests for these builds too. (It would support both this feature, and testing rust against rustup to make sure we don't get any more broken manifests on nightly)

[brson]

I think rustup needs some more support for installing non-channel builds, and the rust ci needs more support for deploying non-channel builds, for multiple reasons. I am not sure what solution I want for that though, and suspect that just teaching rustup to download that tarball by url is not sufficient.

For my purposes, I need to be able to run experimental commits through Rust's release builders and have them deploy to somewhere, so that cargobomb can find them via rustup. For that I might expect there to be a dedicated branch that people can submit PRs to in order to deploy a build, perhaps with most platforms turned off. Those builds get fed into RCS to deploy a non-signed build to a special location.

None of the releases that come out of this system should be mistaken for official releases. I would not put them in the same location w/in static.rust-lang.org where they will pollute our directory structure with unofficial junk.

I would make rustup print big warnings about using any of these releases, especially since rustup doesn't actually deal with sigs today.

There would need to be some new toolchain naming scheme. I'd be inclined to say something like "dev-$sha", except that wouldn't accommodate "master-alt", which is just a different build of the same commit.

I am not enthusiastic at all about advertising "master-alt" - having a secret unofficial nightly channel that those in the know use to get a "fast" nightly is not desirable to me. If nighties should be fast we should make them fast. If we really need to give our users two completely different releases of nightly then I might prefer to give it a more considered design. These builds exist today as a hack for servo.

[SimonSapin]

These builds exist today as a hack for servo.

This sounds wrong to me, even though these builds do make my life easier and I am grateful to everyone who worked on making them available. If Servo can have nice things, everybody should have access to them as well. This thread is about making them easier to get.

run experimental commits through Rust's release builders […] a dedicated branch that people can submit PRs to in order to deploy a build

What that does sound useful and could potentially use a shared more general mechanism, that mechanism doesn’t exist yet and doesn’t seem necessary for getting the "alt" builds that already exist.

If nighties should be fast we should make them fast.

I’d be fine with switching Servo back to nightly (letting go of "test fixes as soon as they’re merged") if we’re willing to disable LLVM assertions in nightly. I’ll start an internals thread about that.

[SimonSapin]

I’ll start an internals thread about that.

https://internals.rust-lang.org/t/disabling-llvm-assertions-in-nightly-builds/5388

[SimonSapin]

What is desired here is mostly to make "alt" toolchains available with rustup somehow. If they are addressed by commit hash (as master-alt-*) and available for every merged Rust PR, then we might as well also make "non-alt" master-* available for every PR too.

Another option is to add nightly-alt-*, addressed by date like nightly-*, and not add anything addressed by commit hash. Internally the full hash is still needed to construct the tarball URLs, but these can be obtained from the GitHub API from a short hash (the sha JSON key in https://api.github.com/repos/rust-lang/rust/commits/{short_hash}), which in turn can be extracted from the Nightly channel manifest (pkg.rustc.version key in https://static.rust-lang.org/dist/{date}/channel-rust-nightly.toml). Or the manifest could be extended to include the full hash, so that the GitHub API is not needed.

[shinglyu]

Any update or decision on this?

[SimonSapin]

rust-lang/rust#44218 is a first step.

[SimonSapin]

rust-lang/rust#45810 remove the compile-time incentive to use alternate builds, and https://internals.rust-lang.org/t/public-stable-rust-services/6072 clarifies that CI builds are not intended for “public consumption”. So this issue is obsolete.