From bd4907d534a1894e28a1f3cf672022e29eabf194 Mon Sep 17 00:00:00 2001 From: Sunjay Varma Date: Sat, 7 Oct 2017 21:53:43 -0400 Subject: [PATCH 1/3] Documenting the process for when rustfmt/rls breakk because of your changes --- CONTRIBUTING.md | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index a86742d7bd4ff..c6c62a8c7b48a 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -363,6 +363,44 @@ outside the submodule. It can also be more convenient during development to set `submodules = false` in the `config.toml` to prevent `x.py` from resetting to the original branch. +#### Breaking rustfmt or rls + +Rust's build system also builds the +[RLS](https://github.com/rust-lang-nursery/rls) +and [rustfmt](https://github.com/rust-lang-nursery/rustfmt). If these tools +break because of your changes, you may run into a sort of "chicken and egg" +problem. Both tools rely on the latest compiler to be built so you can't update +them until the changes you are making to the compiler land. In the meantime, you +can't land your changes to the compiler because the build won't pass until those +tools are fixed. + +That means that, in the default state, you can't update the compiler without +fixing rustfmt and rls first. + +When this happens, follow these steps: + +1. First, if it doesn't exist already, create a `config.toml` by copying + `config.toml.example` in the root directory of the Rust repository. + Set `submodules = false` in the `[build]` section. This will prevent `x.py` + from resetting to the original branch after you make your changes. +2. Run `./x.py test src/tools/rustfmt`. Fix any errors in the submodule itself + (the `src/tools/rustfmt` directory) until it works. +3. Run `./x.py test src/tools/rls`. Fix any errors in the submodule itself + (the `src/tools/rls` directory) until it works. +4. Make a commit for `rustfmt`, if necessary, and send a PR to the master + branch of rust-lang-nursery/rustfmt +5. Do the same, if necessary for the RLS +6. A maintainer of rls/rustfmt will **not** merge the PR. The PR can't be + merged because CI will be broken. Instead a new branch will be created, + and the PR will be pushed to the branch (the PR is left open) +7. On your branch, update the rls/rustfmt submodules to these branches +8. Commit the changes, update your PR to rust-lang/rust +9. Wait for the branch to merge +10. Wait for a nightly +11. A maintainer of rls/rustfmt will merge the original PRs to rls/rustfmt +12. Eventually the rls/rustfmt submodules will get re-updated back to the + master branch + ## Writing Documentation [writing-documentation]: #writing-documentation From 3f90c3a2cfb458c94d514345e701696d74d988c8 Mon Sep 17 00:00:00 2001 From: Sunjay Varma Date: Tue, 10 Oct 2017 16:31:43 -0400 Subject: [PATCH 2/3] Added a section about updating submodules The process for updating rustfmt is quite involved because of the way everything is configured. This section covers the steps for updating rustfmt and rationale behind them. --- CONTRIBUTING.md | 64 ++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 63 insertions(+), 1 deletion(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index c6c62a8c7b48a..34789f144f4b0 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -364,6 +364,7 @@ It can also be more convenient during development to set `submodules = false` in the `config.toml` to prevent `x.py` from resetting to the original branch. #### Breaking rustfmt or rls +[breaking-rustfmt-or-rls]: #breaking-rustfmt-or-rls Rust's build system also builds the [RLS](https://github.com/rust-lang-nursery/rls) @@ -382,7 +383,9 @@ When this happens, follow these steps: 1. First, if it doesn't exist already, create a `config.toml` by copying `config.toml.example` in the root directory of the Rust repository. Set `submodules = false` in the `[build]` section. This will prevent `x.py` - from resetting to the original branch after you make your changes. + from resetting to the original branch after you make your changes. If you + need to [update any submodules to their latest versions][updating-submodules], + see the section of this file about that for more information. 2. Run `./x.py test src/tools/rustfmt`. Fix any errors in the submodule itself (the `src/tools/rustfmt` directory) until it works. 3. Run `./x.py test src/tools/rls`. Fix any errors in the submodule itself @@ -401,6 +404,65 @@ When this happens, follow these steps: 12. Eventually the rls/rustfmt submodules will get re-updated back to the master branch +#### Updating submodules +[updating-submodules]: #updating-submodules + +These instructions are specific to updating `rustfmt`, however they may apply +to the other submodules as well. Please help by improving these instructions +if you find any discrepencies or special cases that need to be addressed. + +To update the `rustfmt` submodule, start by running the appropriate +[`git submodule` command](https://git-scm.com/book/en/v2/Git-Tools-Submodules). +For example, to update to the latest commit on the remote master branch, +you may want to run: +``` +git submodule update --remote src/tools/rustfmt +``` +If you run `./x.py build` now, and you are lucky, it may just work. If you see +an error message about patches that did not resolve to any crates, you will need +to complete a few more steps which are outlined with their rationale below. + +*(This error may change in the future to include more information.)* +``` +error: failed to resolve patches for `https://github.com/rust-lang-nursery/rustfmt` + +Caused by: + patch for `rustfmt-nightly` in `https://github.com/rust-lang-nursery/rustfmt` did not resolve to any crates +failed to run: ~/rust/build/x86_64-unknown-linux-gnu/stage0/bin/cargo build --manifest-path ~/rust/src/bootstrap/Cargo.toml +``` + +If you haven't used the `[patch]` +section of `Cargo.toml` before, there is [some relevant documentation about it +in the cargo docs](http://doc.crates.io/manifest.html#the-patch-section). In +addition to that, you should read the +[Overriding dependencies](http://doc.crates.io/specifying-dependencies.html#overriding-dependencies) +section of the documentation as well. + +Specifically, the following [section in Overriding dependencies](http://doc.crates.io/specifying-dependencies.html#testing-a-bugfix) reveals what the problem is: + +> Next up we need to ensure that our lock file is updated to use this new version of uuid so our project uses the locally checked out copy instead of one from crates.io. The way [patch] works is that it'll load the dependency at ../path/to/uuid and then whenever crates.io is queried for versions of uuid it'll also return the local version. +> +> This means that the version number of the local checkout is significant and will affect whether the patch is used. Our manifest declared uuid = "1.0" which means we'll only resolve to >= 1.0.0, < 2.0.0, and Cargo's greedy resolution algorithm also means that we'll resolve to the maximum version within that range. Typically this doesn't matter as the version of the git repository will already be greater or match the maximum version published on crates.io, but it's important to keep this in mind! + +This says that when we updated the submodule, the version number in our +`src/tools/rustfmt/Cargo.toml` changed. The new version is different from +the version in `Cargo.lock`, so the build can no longer continue. + +To resolve this, we need to update `Cargo.lock`. Luckily, cargo provides a +command to do this easily. + +First, go into the `src/` directory since that is where `Cargo.toml` is in +the rust repository. Then run, `cargo update -p rustfmt-nightly` to solve +the problem. + +``` +$ cd src +$ cargo update -p rustfmt-nightly +``` + +This should change the version listed in `src/Cargo.lock` to the new version you updated +the submodule to. Running `./x.py build` should work now. + ## Writing Documentation [writing-documentation]: #writing-documentation From 790604adad9fd02b92c59d1f937edb902a58b036 Mon Sep 17 00:00:00 2001 From: Sunjay Varma Date: Tue, 17 Oct 2017 22:51:10 -0400 Subject: [PATCH 3/3] Updating the instructions for when a tool breaks to use the new toolstate feature --- CONTRIBUTING.md | 70 ++++++++++++++++++++++++++++--------------------- 1 file changed, 40 insertions(+), 30 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 34789f144f4b0..0f6cba7a95e79 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -363,46 +363,56 @@ outside the submodule. It can also be more convenient during development to set `submodules = false` in the `config.toml` to prevent `x.py` from resetting to the original branch. -#### Breaking rustfmt or rls -[breaking-rustfmt-or-rls]: #breaking-rustfmt-or-rls +#### Breaking Tools Built With The Compiler +[breaking-tools-built-with-the-compiler]: #breaking-tools-built-with-the-compiler -Rust's build system also builds the -[RLS](https://github.com/rust-lang-nursery/rls) -and [rustfmt](https://github.com/rust-lang-nursery/rustfmt). If these tools +Rust's build system builds a number of tools that make use of the +internals of the compiler. This includes clippy, +[RLS](https://github.com/rust-lang-nursery/rls) and +[rustfmt](https://github.com/rust-lang-nursery/rustfmt). If these tools break because of your changes, you may run into a sort of "chicken and egg" -problem. Both tools rely on the latest compiler to be built so you can't update -them until the changes you are making to the compiler land. In the meantime, you -can't land your changes to the compiler because the build won't pass until those -tools are fixed. +problem. These tools rely on the latest compiler to be built so you can't update +them to reflect your changes to the compiler until those changes are merged into +the compiler. At the same time, you can't get your changes merged into the compiler +because the rust-lang/rust build won't pass until those tools build and pass their +tests. -That means that, in the default state, you can't update the compiler without -fixing rustfmt and rls first. +That means that, in the default state, you can't update the compiler without first +fixing rustfmt, rls and the other tools that the compiler builds. -When this happens, follow these steps: +Luckily, a feature was [added to Rust's build](https://github.com/rust-lang/rust/pull/45243) +to make all of this easy to handle. The idea is that you mark the tools as "broken", +so that the rust-lang/rust build passes without trying to build them, then land the change +in the compiler, wait for a nightly, and go update the tools that you broke. Once you're done +and the tools are working again, you go back in the compiler and change the tools back +from "broken". -1. First, if it doesn't exist already, create a `config.toml` by copying +This should avoid a bunch of synchronization dances and is also much easier on contributors as +there's no need to block on rls/rustfmt/other tools changes going upstream. + +Here are those same steps in detail: + +1. (optional) First, if it doesn't exist already, create a `config.toml` by copying `config.toml.example` in the root directory of the Rust repository. Set `submodules = false` in the `[build]` section. This will prevent `x.py` from resetting to the original branch after you make your changes. If you need to [update any submodules to their latest versions][updating-submodules], see the section of this file about that for more information. -2. Run `./x.py test src/tools/rustfmt`. Fix any errors in the submodule itself - (the `src/tools/rustfmt` directory) until it works. -3. Run `./x.py test src/tools/rls`. Fix any errors in the submodule itself - (the `src/tools/rls` directory) until it works. -4. Make a commit for `rustfmt`, if necessary, and send a PR to the master - branch of rust-lang-nursery/rustfmt -5. Do the same, if necessary for the RLS -6. A maintainer of rls/rustfmt will **not** merge the PR. The PR can't be - merged because CI will be broken. Instead a new branch will be created, - and the PR will be pushed to the branch (the PR is left open) -7. On your branch, update the rls/rustfmt submodules to these branches -8. Commit the changes, update your PR to rust-lang/rust -9. Wait for the branch to merge -10. Wait for a nightly -11. A maintainer of rls/rustfmt will merge the original PRs to rls/rustfmt -12. Eventually the rls/rustfmt submodules will get re-updated back to the - master branch +2. (optional) Run `./x.py test src/tools/rustfmt` (substituting the submodule + that broke for `rustfmt`). Fix any errors in the submodule (and possibly others). +3. (optional) Make commits for your changes and send them to upstream repositories as a PR. +4. (optional) Maintainers of these submodules will **not** merge the PR. The PR can't be + merged because CI will be broken. You'll want to write a message on the PR referencing + your change, and how the PR should be merged once your change makes it into a nightly. +5. Update `src/tools/toolstate.toml` to indicate that the tool in question is "broken", + that will disable building it on CI. See the documentation in that file for the exact + configuration values you can use. +6. Commit the changes to `src/tools/toolstate.toml`, **do not update submodules in your commit**, + and then update the PR you have for rust-lang/rust. +7. Wait for your PR to merge. +8. Wait for a nightly +9. (optional) Help land your PR on the upstream repository now that your changes are in nightly. +10. (optional) Send a PR to rust-lang/rust updating the submodule, reverting `src/tools/toolstate.toml` back to a "building" or "testing" state. #### Updating submodules [updating-submodules]: #updating-submodules